KuberHealthy技术分析

一、介绍

  1. 简要介绍

KubeHealthy 是一个用于 Kubernetes 集群的开源工具,用于监控和评估集群中的应用程序和组件的健康状态。它旨在帮助 Kubernetes 用户确保其应用程序在集群中正常运行,并提供有关应用程序和集群健康状态的可视化和报告。

  1. 应用场景

(1)Kubernetes 应用健康监控: KubeHealthy 可用于监控运行在 Kubernetes 集群中的应用程序和微服务的健康状态。它可以定期运行各种健康检查,以确保应用程序的各个组件正常工作,如容器、服务、数据库连接等。

(2)故障检测和通知: KubeHealthy 具备故障检测功能,能够检测应用程序或集群组件的异常状态。当发现问题时,它可以触发警报和通知,以通知管理员或运维团队,帮助他们快速响应和解决问题。

(3)性能优化和容量规划: 通过持续监控应用程序的健康状态,KubeHealthy 可以帮助您了解应用程序的性能瓶颈,并支持容量规划决策。它可以帮助您确定何时需要增加资源,以确保应用程序能够满足预期的性能需求。

二、架构原理

工作原理

以下说明了 Kuberhealthy 如何配置和操作检查器 Pod。流程如下图所示:

  • An admin creates a ​​KuberhealthyCheck​​ resource that calls for a synthetic Kubernetes daemonset to be deployed and tested every 15 minutes. This will ensure that all nodes in the Kubernetes cluster can provision containers properly. 管理员创建一个 KuberhealthyCheck 资源,该资源要求每 15 分钟部署和测试一个合成 Kubernetes 守护进程集。这将确保 Kubernetes 集群中的所有节点都可以正确配置容器。
  • Kuberhealthy observes this new ​​KuberhealthyCheck​​ resource. Kuberhealthy 观察这个新的 KuberhealthyCheck 资源。
  • Kuberhealthy schedules a checker pod to manage the lifecycle of this check. Kuberhealthy 安排一个检查器 pod 来管理此检查的生命周期。
  • The checker pod creates a daemonset using the Kubernetes API. 检查器 Pod 使用 Kubernetes API 创建一个守护进程集。
  • The checker pod observes the daemonset and waits for all daemonset pods to become ​​Ready​checker pod 观察 daemonset 并等待所有 daemonset pod 变为 Ready
  • The checker pod deletes the daemonset using the Kubernetes API. 检查器 pod 使用 Kubernetes API 删除守护进程集。
  • The checker pod observes the daemonset being fully cleaned up and removed. 检查器 pod 观察到守护进程集已被完全清理和删除。
  • The checker pod reports a successful test result back to Kuberhealthy's API. 检查器 Pod 向 Kuberhealthy 的 API 报告成功的测试结果。
  • Kuberhealthy stores this check's state and makes it available to various metrics systems. Kuberhealthy 存储此检查的状态并使其可用于各种指标系统。

三、使用指南

使用软硬环境

硬件要求

无特殊要求,KubeHealthy 主要运行在 Kubernetes 集群中,只要可以正常运行k8s集群即可。

实验硬件

类型

实验使用

服务器

【注:有其他特殊要求的请在列表下面添加行,如内存: 不小于16G】

软件要求

依赖软件列表

类型

要求版本

Kubernetes 或 Helm

Kubernetes1.16 或更高版本 或 Helm3或更高版本

实验软件版本

类型

实验版本

Kubernetes

v1.21.6

操作系统

ubuntu20.04

安装部署

(1)使用YAML文件安装

注:yaml文件位于https://github.com/Comcast/kuberhealthy/blob/master/deploy 下

# If you don't use Prometheus:
# 没有使用Prometheus
# 1 在Kubernetes集群/上下文中创建命名空间“kuberhealthy”
kubectl create namespace kuberhealthy
# 2 部署kuberhealthy 
kubectl apply -f https://raw.githubusercontent.com/kuberhealthy/kuberhealthy/master/deploy/kuberhealthy.yaml

# If you use Prometheus, but not with Prometheus Operator:
# 使用Prometheus,但没使用Prometheus Operator
# 1 在Kubernetes集群/上下文中创建命名空间“kuberhealthy”
kubectl create namespace kuberhealthy
# 2 部署kuberhealthy
kubectl apply -f https://raw.githubusercontent.com/kuberhealthy/kuberhealthy/master/deploy/kuberhealthy-prometheus.yaml

# If you use Prometheus Operator:
# 使用Prometheus Operator
# 1 在Kubernetes集群/上下文中创建命名空间“kuberhealthy”
kubectl create namespace kuberhealthy
# 2 部署kuberhealthy
kubectl apply -f https://raw.githubusercontent.com/kuberhealthy/kuberhealthy/master/deploy/kuberhealthy-prometheus-operator.yaml

3个YAML文件如下:

(1)没有使用Prometheus

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.2
  creationTimestamp: null
  name: khchecks.comcast.github.io
spec:
  group: comcast.github.io
  names:
    kind: KuberhealthyCheck
    listKind: KuberhealthyCheckList
    plural: khchecks
    shortNames:
    - khc
    singular: khcheck
  scope: Namespaced
  preserveUnknownFields: false
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: KuberhealthyCheck represents the data in the CRD for configuring
          an external check for Kuberhealthy
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: Spec holds the desired state of the KuberhealthyCheck (from
              the client).
            properties:
              extraAnnotations:
                additionalProperties:
                  type: string
                type: object
              extraLabels:
                additionalProperties:
                  type: string
                type: object
              podSpec:
                description: PodSpec is a description of a pod.
                properties:
                  activeDeadlineSeconds:
                    description: Optional duration in seconds the pod may be active
                      on the node relative to StartTime before the system will actively
                      try to mark it failed and kill associated containers. Value
                      must be a positive integer.
                    format: int64
                    type: integer
                  affinity:
                    description: If specified, the pod's scheduling constraints
                    properties:
                      nodeAffinity:
                        description: Describes node affinity scheduling rules for
                          the pod.
                        properties:
                          preferredDuringSchedulingIgnoredDuringExecution:
                            description: The scheduler will prefer to schedule pods
                              to nodes that satisfy the affinity expressions specified
                              by this field, but it may choose a node that violates
                              one or more of the expressions. The node that is most
                              preferred is the one with the greatest sum of weights,
                              i.e. for each node that meets all of the scheduling
                              requirements (resource request, requiredDuringScheduling
                              affinity expressions, etc.), compute a sum by iterating
                              through the elements of this field and adding "weight"
                              to the sum if the node matches the corresponding matchExpressions;
                              the node(s) with the highest sum are the most preferred.
                            items:
                              description: An empty preferred scheduling term matches
                                all objects with implicit weight 0 (i.e. it's a no-op).
                                A null preferred scheduling term matches no objects
                                (i.e. is also a no-op).
                              properties:
                                preference:
                                  description: A node selector term, associated with
                                    the corresponding weight.
                                  properties:
                                    matchExpressions:
                                      description: A list of node selector requirements
                                        by node's labels.
                                      items:
                                        description: A node selector requirement is
                                          a selector that contains values, a key,
                                          and an operator that relates the key and
                                          values.
                                        properties:
                                          key:
                                            description: The label key that the selector
                                              applies to.
                                            type: string
                                          operator:
                                            description: Represents a key's relationship
                                              to a set of values. Valid operators
                                              are In, NotIn, Exists, DoesNotExist.
                                              Gt, and Lt.
                                            type: string
                                          values:
                                            description: An array of string values.
                                              If the operator is In or NotIn, the
                                              values array must be non-empty. If the
                                              operator is Exists or DoesNotExist,
                                              the values array must be empty. If the
                                              operator is Gt or Lt, the values array
                                              must have a single element, which will
                                              be interpreted as an integer. This array
                                              is replaced during a strategic merge
                                              patch.
                                            items:
                                              type: string
                                            type: array
                                        required:
                                        - key
                                        - operator
                                        type: object
                                      type: array
                                    matchFields:
                                      description: A list of node selector requirements
                                        by node's fields.
                                      items:
                                        description: A node selector requirement is
                                          a selector that contains values, a key,
                                          and an operator that relates the key and
                                          values.
                                        properties:
                                          key:
                                            description: The label key that the selector
                                              applies to.
                                            type: string
                                          operator:
                                            description: Represents a key's relationship
                                              to a set of values. Valid operators
                                              are In, NotIn, Exists, DoesNotExist.
                                              Gt, and Lt.
                                            type: string
                                          values:
                                            description: An array of string values.
                                              If the operator is In or NotIn, the
                                              values array must be non-empty. If the
                                              operator is Exists or DoesNotExist,
                                              the values array must be empty. If the
                                              operator is Gt or Lt, the values array
                                              must have a single element, which will
                                              be interpreted as an integer. This array
                                              is replaced during a strategic merge
                                              patch.
                                            items:
                                              type: string
                                            type: array
                                        required:
                                        - key
                                        - operator
                                        type: object
                                      type: array
                                  type: object
                                weight:
                                  description: Weight associated with matching the
                                    corresponding nodeSelectorTerm, in the range 1-100.
                                  format: int32
                                  type: integer
                              required:
                              - preference
                              - weight
                              type: object
                            type: array
                          requiredDuringSchedulingIgnoredDuringExecution:
                            description: If the affinity requirements specified by
                              this field are not met at scheduling time, the pod will
                              not be scheduled onto the node. If the affinity requirements
                              specified by this field cease to be met at some point
                              during pod execution (e.g. due to an update), the system
                              may or may not try to eventually evict the pod from
                              its node.
                            properties:
                              nodeSelectorTerms:
                                description: Required. A list of node selector terms.
                                  The terms are ORed.
                                items:
                                  description: A null or empty node selector term
                                    matches no objects. The requirements of them are
                                    ANDed. The TopologySelectorTerm type implements
                                    a subset of the NodeSelectorTerm.
                                  properties:
                                    matchExpressions:
                                      description: A list of node selector requirements
                                        by node's labels.
                                      items:
                                        description: A node selector requirement is
                                          a selector that contains values, a key,
                                          and an operator that relates the key and
                                          values.
                                        properties:
                                          key:
                                            description: The label key that the selector
                                              applies to.
                                            type: string
                                          operator:
                                            description: Represents a key's relationship
                                              to a set of values. Valid operators
                                              are In, NotIn, Exists, DoesNotExist.
                                              Gt, and Lt.
                                            type: string
                                          values:
                                            description: An array of string values.
                                              If the operator is In or NotIn, the
                                              values array must be non-empty. If the
                                              operator is Exists or DoesNotExist,
                                              the values array must be empty. If the
                                              operator is Gt or Lt, the values array
                                              must have a single element, which will
                                              be interpreted as an integer. This array
                                              is replaced during a strategic merge
                                              patch.
                                            items:
                                              type: string
                                            type: array
                                        required:
                                        - key
                                        - operator
                                        type: object
                                      type: array
                                    matchFields:
                                      description: A list of node selector requirements
                                        by node's fields.
                                      items:
                                        description: A node selector requirement is
                                          a selector that contains values, a key,
                                          and an operator that relates the key and
                                          values.
                                        properties:
                                          key:
                                            description: The label key that the selector
                                              applies to.
                                            type: string
                                          operator:
                                            description: Represents a key's relationship
                                              to a set of values. Valid operators
                                              are In, NotIn, Exists, DoesNotExist.
                                              Gt, and Lt.
                                            type: string
                                          values:
                                            description: An array of string values.
                                              If the operator is In or NotIn, the
                                              values array must be non-empty. If the
                                              operator is Exists or DoesNotExist,
                                              the values array must be empty. If the
                                              operator is Gt or Lt, the values array
                                              must have a single element, which will
                                              be interpreted as an integer. This array
                                              is replaced during a strategic merge
                                              patch.
                                            items:
                                              type: string
                                            type: array
                                        required:
                                        - key
                                        - operator
                                        type: object
                                      type: array
                                  type: object
                                type: array
                            required:
                            - nodeSelectorTerms
                            type: object
                        type: object
                      podAffinity:
                        description: Describes pod affinity scheduling rules (e.g.
                          co-locate this pod in the same node, zone, etc. as some
                          other pod(s)).
                        properties:
                          preferredDuringSchedulingIgnoredDuringExecution:
                            description: The scheduler will prefer to schedule pods
                              to nodes that satisfy the affinity expressions specified
                              by this field, but it may choose a node that violates
                              one or more of the expressions. The node that is most
                              preferred is the one with the greatest sum of weights,
                              i.e. for each node that meets all of the scheduling
                              requirements (resource request, requiredDuringScheduling
                              affinity expressions, etc.), compute a sum by iterating
                              through the elements of this field and adding "weight"
                              to the sum if the node has pods which matches the corresponding
                              podAffinityTerm; the node(s) with the highest sum are
                              the most preferred.
                            items:
                              description: The weights of all of the matched WeightedPodAffinityTerm
                                fields are added per-node to find the most preferred
                                node(s)
                              properties:
                                podAffinityTerm:
                                  description: Required. A pod affinity term, associated
                                    with the corresponding weight.
                                  properties:
                                    labelSelector:
                                      description: A label query over a set of resources,
                                        in this case pods.
                                      properties:
                                        matchExpressions:
                                          description: matchExpressions is a list
                                            of label selector requirements. The requirements
                                            are ANDed.
                                          items:
                                            description: A label selector requirement
                                              is a selector that contains values,
                                              a key, and an operator that relates
                                              the key and values.
                                            properties:
                                              key:
                                                description: key is the label key
                                                  that the selector applies to.
                                                type: string
                                              operator:
                                                description: operator represents a
                                                  key's relationship to a set of values.
                                                  Valid operators are In, NotIn, Exists
                                                  and DoesNotExist.
                                                type: string
                                              values:
                                                description: values is an array of
                                                  string values. If the operator is
                                                  In or NotIn, the values array must
                                                  be non-empty. If the operator is
                                                  Exists or DoesNotExist, the values
                                                  array must be empty. This array
                                                  is replaced during a strategic merge
                                                  patch.
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                            - key
                                            - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          description: matchLabels is a map of {key,value}
                                            pairs. A single {key,value} in the matchLabels
                                            map is equivalent to an element of matchExpressions,
                                            whose key field is "key", the operator
                                            is "In", and the values array contains
                                            only "value". The requirements are ANDed.
                                          type: object
                                      type: object
                                    namespaceSelector:
                                      description: A label query over the set of namespaces
                                        that the term applies to. The term is applied
                                        to the union of the namespaces selected by
                                        this field and the ones listed in the namespaces
                                        field. null selector and null or empty namespaces
                                        list means "this pod's namespace". An empty
                                        selector ({}) matches all namespaces. This
                                        field is alpha-level and is only honored when
                                        PodAffinityNamespaceSelector feature is enabled.
                                      properties:
                                        matchExpressions:
                                          description: matchExpressions is a list
                                            of label selector requirements. The requirements
                                            are ANDed.
                                          items:
                                            description: A label selector requirement
                                              is a selector that contains values,
                                              a key, and an operator that relates
                                              the key and values.
                                            properties:
                                              key:
                                                description: key is the label key
                                                  that the selector applies to.
                                                type: string
                                              operator:
                                                description: operator represents a
                                                  key's relationship to a set of values.
                                                  Valid operators are In, NotIn, Exists
                                                  and DoesNotExist.
                                                type: string
                                              values:
                                                description: values is an array of
                                                  string values. If the operator is
                                                  In or NotIn, the values array must
                                                  be non-empty. If the operator is
                                                  Exists or DoesNotExist, the values
                                                  array must be empty. This array
                                                  is replaced during a strategic merge
                                                  patch.
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                            - key
                                            - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          description: matchLabels is a map of {key,value}
                                            pairs. A single {key,value} in the matchLabels
                                            map is equivalent to an element of matchExpressions,
                                            whose key field is "key", the operator
                                            is "In", and the values array contains
                                            only "value". The requirements are ANDed.
                                          type: object
                                      type: object
                                    namespaces:
                                      description: namespaces specifies a static list
                                        of namespace names that the term applies to.
                                        The term is applied to the union of the namespaces
                                        listed in this field and the ones selected
                                        by namespaceSelector. null or empty namespaces
                                        list and null namespaceSelector means "this
                                        pod's namespace"
                                      items:
                                        type: string
                                      type: array
                                    topologyKey:
                                      description: This pod should be co-located (affinity)
                                        or not co-located (anti-affinity) with the
                                        pods matching the labelSelector in the specified
                                        namespaces, where co-located is defined as
                                        running on a node whose value of the label
                                        with key topologyKey matches that of any node
                                        on which any of the selected pods is running.
                                        Empty topologyKey is not allowed.
                                      type: string
                                  required:
                                  - topologyKey
                                  type: object
                                weight:
                                  description: weight associated with matching the
                                    corresponding podAffinityTerm, in the range 1-100.
                                  format: int32
                                  type: integer
                              required:
                              - podAffinityTerm
                              - weight
                              type: object
                            type: array
                          requiredDuringSchedulingIgnoredDuringExecution:
                            description: If the affinity requirements specified by
                              this field are not met at scheduling time, the pod will
                              not be scheduled onto the node. If the affinity requirements
                              specified by this field cease to be met at some point
                              during pod execution (e.g. due to a pod label update),
                              the system may or may not try to eventually evict the
                              pod from its node. When there are multiple elements,
                              the lists of nodes corresponding to each podAffinityTerm
                              are intersected, i.e. all terms must be satisfied.
                            items:
                              description: Defines a set of pods (namely those matching
                                the labelSelector relative to the given namespace(s))
                                that this pod should be co-located (affinity) or not
                                co-located (anti-affinity) with, where co-located
                                is defined as running on a node whose value of the
                                label with key <topologyKey> matches that of any node
                                on which a pod of the set of pods is running
                              properties:
                                labelSelector:
                                  description: A label query over a set of resources,
                                    in this case pods.
                                  properties:
                                    matchExpressions:
                                      description: matchExpressions is a list of label
                                        selector requirements. The requirements are
                                        ANDed.
                                      items:
                                        description: A label selector requirement
                                          is a selector that contains values, a key,
                                          and an operator that relates the key and
                                          values.
                                        properties:
                                          key:
                                            description: key is the label key that
                                              the selector applies to.
                                            type: string
                                          operator:
                                            description: operator represents a key's
                                              relationship to a set of values. Valid
                                              operators are In, NotIn, Exists and
                                              DoesNotExist.
                                            type: string
                                          values:
                                            description: values is an array of string
                                              values. If the operator is In or NotIn,
                                              the values array must be non-empty.
                                              If the operator is Exists or DoesNotExist,
                                              the values array must be empty. This
                                              array is replaced during a strategic
                                              merge patch.
                                            items:
                                              type: string
                                            type: array
                                        required:
                                        - key
                                        - operator
                                        type: object
                                      type: array
                                    matchLabels:
                                      additionalProperties:
                                        type: string
                                      description: matchLabels is a map of {key,value}
                                        pairs. A single {key,value} in the matchLabels
                                        map is equivalent to an element of matchExpressions,
                                        whose key field is "key", the operator is
                                        "In", and the values array contains only "value".
                                        The requirements are ANDed.
                                      type: object
                                  type: object
                                namespaceSelector:
                                  description: A label query over the set of namespaces
                                    that the term applies to. The term is applied
                                    to the union of the namespaces selected by this
                                    field and the ones listed in the namespaces field.
                                    null selector and null or empty namespaces list
                                    means "this pod's namespace". An empty selector
                                    ({}) matches all namespaces. This field is alpha-level
                                    and is only honored when PodAffinityNamespaceSelector
                                    feature is enabled.
                                  properties:
                                    matchExpressions:
                                      description: matchExpressions is a list of label
                                        selector requirements. The requirements are
                                        ANDed.
                                      items:
                                        description: A label selector requirement
                                          is a selector that contains values, a key,
                                          and an operator that relates the key and
                                          values.
                                        properties:
                                          key:
                                            description: key is the label key that
                                              the selector applies to.
                                            type: string
                                          operator:
                                            description: operator represents a key's
                                              relationship to a set of values. Valid
                                              operators are In, NotIn, Exists and
                                              DoesNotExist.
                                            type: string
                                          values:
                                            description: values is an array of string
                                              values. If the operator is In or NotIn,
                                              the values array must be non-empty.
                                              If the operator is Exists or DoesNotExist,
                                              the values array must be empty. This
                                              array is replaced during a strategic
                                              merge patch.
                                            items:
                                              type: string
                                            type: array
                                        required:
                                        - key
                                        - operator
                                        type: object
                                      type: array
                                    matchLabels:
                                      additionalProperties:
                                        type: string
                                      description: matchLabels is a map of {key,value}
                                        pairs. A single {key,value} in the matchLabels
                                        map is equivalent to an element of matchExpressions,
                                        whose key field is "key", the operator is
                                        "In", and the values array contains only "value".
                                        The requirements are ANDed.
                                      type: object
                                  type: object
                                namespaces:
                                  description: namespaces specifies a static list
                                    of namespace names that the term applies to. The
                                    term is applied to the union of the namespaces
                                    listed in this field and the ones selected by
                                    namespaceSelector. null or empty namespaces list
                                    and null namespaceSelector means "this pod's namespace"
                                  items:
                                    type: string
                                  type: array
                                topologyKey:
                                  description: This pod should be co-located (affinity)
                                    or not co-located (anti-affinity) with the pods
                                    matching the labelSelector in the specified namespaces,
                                    where co-located is defined as running on a node
                                    whose value of the label with key topologyKey
                                    matches that of any node on which any of the selected
                                    pods is running. Empty topologyKey is not allowed.
                                  type: string
                              required:
                              - topologyKey
                              type: object
                            type: array
                        type: object
                      podAntiAffinity:
                        description: Describes pod anti-affinity scheduling rules
                          (e.g. avoid putting this pod in the same node, zone, etc.
                          as some other pod(s)).
                        properties:
                          preferredDuringSchedulingIgnoredDuringExecution:
                            description: The scheduler will prefer to schedule pods
                              to nodes that satisfy the anti-affinity expressions
                              specified by this field, but it may choose a node that
                              violates one or more of the expressions. The node that
                              is most preferred is the one with the greatest sum of
                              weights, i.e. for each node that meets all of the scheduling
                              requirements (resource request, requiredDuringScheduling
                              anti-affinity expressions, etc.), compute a sum by iterating
                              through the elements of this field and adding "weight"
                              to the sum if the node has pods which matches the corresponding
                              podAffinityTerm; the node(s) with the highest sum are
                              the most preferred.
                            items:
                              description: The weights of all of the matched WeightedPodAffinityTerm
                                fields are added per-node to find the most preferred
                                node(s)
                              properties:
                                podAffinityTerm:
                                  description: Required. A pod affinity term, associated
                                    with the corresponding weight.
                                  properties:
                                    labelSelector:
                                      description: A label query over a set of resources,
                                        in this case pods.
                                      properties:
                                        matchExpressions:
                                          description: matchExpressions is a list
                                            of label selector requirements. The requirements
                                            are ANDed.
                                          items:
                                            description: A label selector requirement
                                              is a selector that contains values,
                                              a key, and an operator that relates
                                              the key and values.
                                            properties:
                                              key:
                                                description: key is the label key
                                                  that the selector applies to.
                                                type: string
                                              operator:
                                                description: operator represents a
                                                  key's relationship to a set of values.
                                                  Valid operators are In, NotIn, Exists
                                                  and DoesNotExist.
                                                type: string
                                              values:
                                                description: values is an array of
                                                  string values. If the operator is
                                                  In or NotIn, the values array must
                                                  be non-empty. If the operator is
                                                  Exists or DoesNotExist, the values
                                                  array must be empty. This array
                                                  is replaced during a strategic merge
                                                  patch.
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                            - key
                                            - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          description: matchLabels is a map of {key,value}
                                            pairs. A single {key,value} in the matchLabels
                                            map is equivalent to an element of matchExpressions,
                                            whose key field is "key", the operator
                                            is "In", and the values array contains
                                            only "value". The requirements are ANDed.
                                          type: object
                                      type: object
                                    namespaceSelector:
                                      description: A label query over the set of namespaces
                                        that the term applies to. The term is applied
                                        to the union of the namespaces selected by
                                        this field and the ones listed in the namespaces
                                        field. null selector and null or empty namespaces
                                        list means "this pod's namespace". An empty
                                        selector ({}) matches all namespaces. This
                                        field is alpha-level and is only honored when
                                        PodAffinityNamespaceSelector feature is enabled.
                                      properties:
                                        matchExpressions:
                                          description: matchExpressions is a list
                                            of label selector requirements. The requirements
                                            are ANDed.
                                          items:
                                            description: A label selector requirement
                                              is a selector that contains values,
                                              a key, and an operator that relates
                                              the key and values.
                                            properties:
                                              key:
                                                description: key is the label key
                                                  that the selector applies to.
                                                type: string
                                              operator:
                                                description: operator represents a
                                                  key's relationship to a set of values.
                                                  Valid operators are In, NotIn, Exists
                                                  and DoesNotExist.
                                                type: string
                                              values:
                                                description: values is an array of
                                                  string values. If the operator is
                                                  In or NotIn, the values array must
                                                  be non-empty. If the operator is
                                                  Exists or DoesNotExist, the values
                                                  array must be empty. This array
                                                  is replaced during a strategic merge
                                                  patch.
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                            - key
                                            - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          description: matchLabels is a map of {key,value}
                                            pairs. A single {key,value} in the matchLabels
                                            map is equivalent to an element of matchExpressions,
                                            whose key field is "key", the operator
                                            is "In", and the values array contains
                                            only "value". The requirements are ANDed.
                                          type: object
                                      type: object
                                    namespaces:
                                      description: namespaces specifies a static list
                                        of namespace names that the term applies to.
                                        The term is applied to the union of the namespaces
                                        listed in this field and the ones selected
                                        by namespaceSelector. null or empty namespaces
                                        list and null namespaceSelector means "this
                                        pod's namespace"
                                      items:
                                        type: string
                                      type: array
                                    topologyKey:
                                      description: This pod should be co-located (affinity)
                                        or not co-located (anti-affinity) with the
                                        pods matching the labelSelector in the specified
                                        namespaces, where co-located is defined as
                                        running on a node whose value of the label
                                        with key topologyKey matches that of any node
                                        on which any of the selected pods is running.
                                        Empty topologyKey is not allowed.
                                      type: string
                                  required:
                                  - topologyKey
                                  type: object
                                weight:
                                  description: weight associated with matching the
                                    corresponding podAffinityTerm, in the range 1-100.
                                  format: int32
                                  type: integer
                              required:
                              - podAffinityTerm
                              - weight
                              type: object
                            type: array
                          requiredDuringSchedulingIgnoredDuringExecution:
                            description: If the anti-affinity requirements specified
                              by this field are not met at scheduling time, the pod
                              will not be scheduled onto the node. If the anti-affinity
                              requirements specified by this field cease to be met
                              at some point during pod execution (e.g. due to a pod
                              label update), the system may or may not try to eventually
                              evict the pod from its node. When there are multiple
                              elements, the lists of nodes corresponding to each podAffinityTerm
                              are intersected, i.e. all terms must be satisfied.
                            items:
                              description: Defines a set of pods (namely those matching
                                the labelSelector relative to the given namespace(s))
                                that this pod should be co-located (affinity) or not
                                co-located (anti-affinity) with, where co-located
                                is defined as running on a node whose value of the
                                label with key <topologyKey> matches that of any node
                                on which a pod of the set of pods is running
                              properties:
                                labelSelector:
                                  description: A label query over a set of resources,
                                    in this case pods.
                                  properties:
                                    matchExpressions:
                                      description: matchExpressions is a list of label
                                        selector requirements. The requirements are
                                        ANDed.
                                      items:
                                        description: A label selector requirement
                                          is a selector that contains values, a key,
                                          and an operator that relates the key and
                                          values.
                                        properties:
                                          key:
                                            description: key is the label key that
                                              the selector applies to.
                                            type: string
                                          operator:
                                            description: operator represents a key's
                                              relationship to a set of values. Valid
                                              operators are In, NotIn, Exists and
                                              DoesNotExist.
                                            type: string
                                          values:
                                            description: values is an array of string
                                              values. If the operator is In or NotIn,
                                              the values array must be non-empty.
                                              If the operator is Exists or DoesNotExist,
                                              the values array must be empty. This
                                              array is replaced during a strategic
                                              merge patch.
                                            items:
                                              type: string
                                            type: array
                                        required:
                                        - key
                                        - operator
                                        type: object
                                      type: array
                                    matchLabels:
                                      additionalProperties:
                                        type: string
                                      description: matchLabels is a map of {key,value}
                                        pairs. A single {key,value} in the matchLabels
                                        map is equivalent to an element of matchExpressions,
                                        whose key field is "key", the operator is
                                        "In", and the values array contains only "value".
                                        The requirements are ANDed.
                                      type: object
                                  type: object
                                namespaceSelector:
                                  description: A label query over the set of namespaces
                                    that the term applies to. The term is applied
                                    to the union of the namespaces selected by this
                                    field and the ones listed in the namespaces field.
                                    null selector and null or empty namespaces list
                                    means "this pod's namespace". An empty selector
                                    ({}) matches all namespaces. This field is alpha-level
                                    and is only honored when PodAffinityNamespaceSelector
                                    feature is enabled.
                                  properties:
                                    matchExpressions:
                                      description: matchExpressions is a list of label
                                        selector requirements. The requirements are
                                        ANDed.
                                      items:
                                        description: A label selector requirement
                                          is a selector that contains values, a key,
                                          and an operator that relates the key and
                                          values.
                                        properties:
                                          key:
                                            description: key is the label key that
                                              the selector applies to.
                                            type: string
                                          operator:
                                            description: operator represents a key's
                                              relationship to a set of values. Valid
                                              operators are In, NotIn, Exists and
                                              DoesNotExist.
                                            type: string
                                          values:
                                            description: values is an array of string
                                              values. If the operator is In or NotIn,
                                              the values array must be non-empty.
                                              If the operator is Exists or DoesNotExist,
                                              the values array must be empty. This
                                              array is replaced during a strategic
                                              merge patch.
                                            items:
                                              type: string
                                            type: array
                                        required:
                                        - key
                                        - operator
                                        type: object
                                      type: array
                                    matchLabels:
                                      additionalProperties:
                                        type: string
                                      description: matchLabels is a map of {key,value}
                                        pairs. A single {key,value} in the matchLabels
                                        map is equivalent to an element of matchExpressions,
                                        whose key field is "key", the operator is
                                        "In", and the values array contains only "value".
                                        The requirements are ANDed.
                                      type: object
                                  type: object
                                namespaces:
                                  description: namespaces specifies a static list
                                    of namespace names that the term applies to. The
                                    term is applied to the union of the namespaces
                                    listed in this field and the ones selected by
                                    namespaceSelector. null or empty namespaces list
                                    and null namespaceSelector means "this pod's namespace"
                                  items:
                                    type: string
                                  type: array
                                topologyKey:
                                  description: This pod should be co-located (affinity)
                                    or not co-located (anti-affinity) with the pods
                                    matching the labelSelector in the specified namespaces,
                                    where co-located is defined as running on a node
                                    whose value of the label with key topologyKey
                                    matches that of any node on which any of the selected
                                    pods is running. Empty topologyKey is not allowed.
                                  type: string
                              required:
                              - topologyKey
                              type: object
                            type: array
                        type: object
                    type: object
                  automountServiceAccountToken:
                    description: AutomountServiceAccountToken indicates whether a
                      service account token should be automatically mounted.
                    type: boolean
                  containers:
                    description: List of containers belonging to the pod. Containers
                      cannot currently be added or removed. There must be at least
                      one container in a Pod. Cannot be updated.
                    items:
                      description: A single application container that you want to
                        run within a pod.
                      properties:
                        args:
                          description: 'Arguments to the entrypoint. The docker image''s
                            CMD is used if this is not provided. Variable references
                            $(VAR_NAME) are expanded using the container''s environment.
                            If a variable cannot be resolved, the reference in the
                            input string will be unchanged. The $(VAR_NAME) syntax
                            can be escaped with a double $$, ie: $$(VAR_NAME). Escaped
                            references will never be expanded, regardless of whether
                            the variable exists or not. Cannot be updated. More info:
                            https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
                          items:
                            type: string
                          type: array
                        command:
                          description: 'Entrypoint array. Not executed within a shell.
                            The docker image''s ENTRYPOINT is used if this is not
                            provided. Variable references $(VAR_NAME) are expanded
                            using the container''s environment. If a variable cannot
                            be resolved, the reference in the input string will be
                            unchanged. The $(VAR_NAME) syntax can be escaped with
                            a double $$, ie: $$(VAR_NAME). Escaped references will
                            never be expanded, regardless of whether the variable
                            exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
                          items:
                            type: string
                          type: array
                        env:
                          description: List of environment variables to set in the
                            container. Cannot be updated.
                          items:
                            description: EnvVar represents an environment variable
                              present in a Container.
                            properties:
                              name:
                                description: Name of the environment variable. Must
                                  be a C_IDENTIFIER.
                                type: string
                              value:
                                description: 'Variable references $(VAR_NAME) are
                                  expanded using the previous defined environment
                                  variables in the container and any service environment
                                  variables. If a variable cannot be resolved, the
                                  reference in the input string will be unchanged.
                                  The $(VAR_NAME) syntax can be escaped with a double
                                  $$, ie: $$(VAR_NAME). Escaped references will never
                                  be expanded, regardless of whether the variable
                                  exists or not. Defaults to "".'
                                type: string
                              valueFrom:
                                description: Source for the environment variable's
                                  value. Cannot be used if value is not empty.
                                properties:
                                  configMapKeyRef:
                                    description: Selects a key of a ConfigMap.
                                    properties:
                                      key:
                                        description: The key to select.
                                        type: string
                                      name:
                                        description: 'Name of the referent. More info:
                                          https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                                          TODO: Add other useful fields. apiVersion,
                                          kind, uid?'
                                        type: string
                                      optional:
                                        description: Specify whether the ConfigMap
                                          or its key must be defined
                                        type: boolean
                                    required:
                                    - key
                                    type: object
                                  fieldRef:
                                    description: 'Selects a field of the pod: supports
                                      metadata.name, metadata.namespace, `metadata.labels[''<KEY>'']`,
                                      `metadata.annotations[''<KEY>'']`, spec.nodeName,
                                      spec.serviceAccountName, status.hostIP, status.podIP,
                                      status.podIPs.'
                                    properties:
                                      apiVersion:
                                        description: Version of the schema the FieldPath
                                          is written in terms of, defaults to "v1".
                                        type: string
                                      fieldPath:
                                        description: Path of the field to select in
                                          the specified API version.
                                        type: string
                                    required:
                                    - fieldPath
                                    type: object
                                  resourceFieldRef:
                                    description: 'Selects a resource of the container:
                                      only resources limits and requests (limits.cpu,
                                      limits.memory, limits.ephemeral-storage, requests.cpu,
                                      requests.memory and requests.ephemeral-storage)
                                      are currently supported.'
                                    properties:
                                      containerName:
                                        description: 'Container name: required for
                                          volumes, optional for env vars'
                                        type: string
                                      divisor:
                                        anyOf:
                                        - type: integer
                                        - type: string
                                        description: Specifies the output format of
                                          the exposed resources, defaults to "1"
                                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                        x-kubernetes-int-or-string: true
                                      resource:
                                        description: 'Required: resource to select'
                                        type: string
                                    required:
                                    - resource
                                    type: object
                                  secretKeyRef:
                                    description: Selects a key of a secret in the
                                      pod's namespace
                                    properties:
                                      key:
                                        description: The key of the secret to select
                                          from.  Must be a valid secret key.
                                        type: string
                                      name:
                                        description: 'Name of the referent. More info:
                                          https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                                          TODO: Add other useful fields. apiVersion,
                                          kind, uid?'
                                        type: string
                                      optional:
                                        description: Specify whether the Secret or
                                          its key must be defined
                                        type: boolean
                                    required:
                                    - key
                                    type: object
                                type: object
                            required:
                            - name
                            type: object
                          type: array
                        envFrom:
                          description: List of sources to populate environment variables
                            in the container. The keys defined within a source must
                            be a C_IDENTIFIER. All invalid keys will be reported as
                            an event when the container is starting. When a key exists
                            in multiple sources, the value associated with the last
                            source will take precedence. Values defined by an Env
                            with a duplicate key will take precedence. Cannot be updated.
                          items:
                            description: EnvFromSource represents the source of a
                              set of ConfigMaps
                            properties:
                              configMapRef:
                                description: The ConfigMap to select from
                                properties:
                                  name:
                                    description: 'Name of the referent. More info:
                                      https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                                      TODO: Add other useful fields. apiVersion, kind,
                                      uid?'
                                    type: string
                                  optional:
                                    description: Specify whether the ConfigMap must
                                      be defined
                                    type: boolean
                                type: object
                              prefix:
                                description: An optional identifier to prepend to
                                  each key in the ConfigMap. Must be a C_IDENTIFIER.
                                type: string
                              secretRef:
                                description: The Secret to select from
                                properties:
                                  name:
                                    description: 'Name of the referent. More info:
                                      https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                                      TODO: Add other useful fields. apiVersion, kind,
                                      uid?'
                                    type: string
                                  optional:
                                    description: Specify whether the Secret must be
                                      defined
                                    type: boolean
                                type: object
                            type: object
                          type: array
                        image:
                          description: 'Docker image name. More info: https://kubernetes.io/docs/concepts/containers/images
                            This field is optional to allow higher level config management
                            to default or override container images in workload controllers
                            like Deployments and StatefulSets.'
                          type: string
                        imagePullPolicy:
                          description: 'Image pull policy. One of Always, Never, IfNotPresent.
                            Defaults to Always if :latest tag is specified, or IfNotPresent
                            otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images'
                          type: string
                        lifecycle:
                          description: Actions that the management system should take
                            in response to container lifecycle events. Cannot be updated.
                          properties:
                            postStart:
                              description: 'PostStart is called immediately after
                                a container is created. If the handler fails, the
                                container is terminated and restarted according to
                                its restart policy. Other management of the container
                                blocks until the hook completes. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
                              properties:
                                exec:
                                  description: One and only one of the following should
                                    be specified. Exec specifies the action to take.
                                  properties:
                                    command:
                                      description: Command is the command line to
                                        execute inside the container, the working
                                        directory for the command  is root ('/') in
                                        the container's filesystem. The command is
                                        simply exec'd, it is not run inside a shell,
                                        so traditional shell instructions ('|', etc)
                                        won't work. To use a shell, you need to explicitly
                                        call out to that shell. Exit status of 0 is
                                        treated as live/healthy and non-zero is unhealthy.
                                      items:
                                        type: string
                                      type: array
                                  type: object
                                httpGet:
                                  description: HTTPGet specifies the http request
                                    to perform.
                                  properties:
                                    host:
                                      description: Host name to connect to, defaults
                                        to the pod IP. You probably want to set "Host"
                                        in httpHeaders instead.
                                      type: string
                                    httpHeaders:
                                      description: Custom headers to set in the request.
                                        HTTP allows repeated headers.
                                      items:
                                        description: HTTPHeader describes a custom
                                          header to be used in HTTP probes
                                        properties:
                                          name:
                                            description: The header field name
                                            type: string
                                          value:
                                            description: The header field value
                                            type: string
                                        required:
                                        - name
                                        - value
                                        type: object
                                      type: array
                                    path:
                                      description: Path to access on the HTTP server.
                                      type: string
                                    port:
                                      anyOf:
                                      - type: integer
                                      - type: string
                                      description: Name or number of the port to access
                                        on the container. Number must be in the range
                                        1 to 65535. Name must be an IANA_SVC_NAME.
                                      x-kubernetes-int-or-string: true
                                    scheme:
                                      description: Scheme to use for connecting to
                                        the host. Defaults to HTTP.
                                      type: string
                                  required:
                                  - port
                                  type: object
                                tcpSocket:
                                  description: 'TCPSocket specifies an action involving
                                    a TCP port. TCP hooks not yet supported TODO:
                                    implement a realistic TCP lifecycle hook'
                                  properties:
                                    host:
                                      description: 'Optional: Host name to connect
                                        to, defaults to the pod IP.'
                                      type: string
                                    port:
                                      anyOf:
                                      - type: integer
                                      - type: string
                                      description: Number or name of the port to access
                                        on the container. Number must be in the range
                                        1 to 65535. Name must be an IANA_SVC_NAME.
                                      x-kubernetes-int-or-string: true
                                  required:
                                  - port
                                  type: object
                              type: object
                            preStop:
                              description: 'PreStop is called immediately before a
                                container is terminated due to an API request or management
                                event such as liveness/startup probe failure, preemption,
                                resource contention, etc. The handler is not called
                                if the container crashes or exits. The reason for
                                termination is passed to the handler. The Pod''s termination
                                grace period countdown begins before the PreStop hooked
                                is executed. Regardless of the outcome of the handler,
                                the container will eventually terminate within the
                                Pod''s termination grace period. Other management
                                of the container blocks until the hook completes or
                                until the termination grace period is reached. More
                                info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
                              properties:
                                exec:
                                  description: One and only one of the following should
                                    be specified. Exec specifies the action to take.
                                  properties:
                                    command:
                                      description: Command is the command line to
                                        execute inside the container, the working
                                        directory for the command  is root ('/') in
                                        the container's filesystem. The command is
                                        simply exec'd, it is not run inside a shell,
                                        so traditional shell instructions ('|', etc)
                                        won't work. To use a shell, you need to explicitly
                                        call out to that shell. Exit status of 0 is
                                        treated as live/healthy and non-zero is unhealthy.
                                      items:
                                        type: string
                                      type: array
                                  type: object
                                httpGet:
                                  description: HTTPGet specifies the http request
                                    to perform.
                                  properties:
                                    host:
                                      description: Host name to connect to, defaults
                                        to the pod IP. You probably want to set "Host"
                                        in httpHeaders instead.
                                      type: string
                                    httpHeaders:
                                      description: Custom headers to set in the request.
                                        HTTP allows repeated headers.
                                      items:
                                        description: HTTPHeader describes a custom
                                          header to be used in HTTP probes
                                        properties:
                                          name:
                                            description: The header field name
                                            type: string
                                          value:
                                            description: The header field value
                                            type: string
                                        required:
                                        - name
                                        - value
                                        type: object
                                      type: array
                                    path:
                                      description: Path to access on the HTTP server.
                                      type: string
                                    port:
                                      anyOf:
                                      - type: integer
                                      - type: string
                                      description: Name or number of the port to access
                                        on the container. Number must be in the range
                                        1 to 65535. Name must be an IANA_SVC_NAME.
                                      x-kubernetes-int-or-string: true
                                    scheme:
                                      description: Scheme to use for connecting to
                                        the host. Defaults to HTTP.
                                      type: string
                                  required:
                                  - port
                                  type: object
                                tcpSocket:
                                  description: 'TCPSocket specifies an action involving
                                    a TCP port. TCP hooks not yet supported TODO:
                                    implement a realistic TCP lifecycle hook'
                                  properties:
                                    host:
                                      description: 'Optional: Host name to connect
                                        to, defaults to the pod IP.'
                                      type: string
                                    port:
                                      anyOf:
                                      - type: integer
                                      - type: string
                                      description: Number or name of the port to access
                                        on the container. Number must be in the range
                                        1 to 65535. Name must be an IANA_SVC_NAME.
                                      x-kubernetes-int-or-string: true
                                  required:
                                  - port
                                  type: object
                              type: object
                          type: object
                        livenessProbe:
                          description: 'Periodic probe of container liveness. Container
                            will be restarted if the probe fails. Cannot be updated.
                            More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                          properties:
                            exec:
                              description: One and only one of the following should
                                be specified. Exec specifies the action to take.
                              properties:
                                command:
                                  description: Command is the command line to execute
                                    inside the container, the working directory for
                                    the command  is root ('/') in the container's
                                    filesystem. The command is simply exec'd, it is
                                    not run inside a shell, so traditional shell instructions
                                    ('|', etc) won't work. To use a shell, you need
                                    to explicitly call out to that shell. Exit status
                                    of 0 is treated as live/healthy and non-zero is
                                    unhealthy.
                                  items:
                                    type: string
                                  type: array
                              type: object
                            failureThreshold:
                              description: Minimum consecutive failures for the probe
                                to be considered failed after having succeeded. Defaults
                                to 3. Minimum value is 1.
                              format: int32
                              type: integer
                            httpGet:
                              description: HTTPGet specifies the http request to perform.
                              properties:
                                host:
                                  description: Host name to connect to, defaults to
                                    the pod IP. You probably want to set "Host" in
                                    httpHeaders instead.
                                  type: string
                                httpHeaders:
                                  description: Custom headers to set in the request.
                                    HTTP allows repeated headers.
                                  items:
                                    description: HTTPHeader describes a custom header
                                      to be used in HTTP probes
                                    properties:
                                      name:
                                        description: The header field name
                                        type: string
                                      value:
                                        description: The header field value
                                        type: string
                                    required:
                                    - name
                                    - value
                                    type: object
                                  type: array
                                path:
                                  description: Path to access on the HTTP server.
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Name or number of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                                scheme:
                                  description: Scheme to use for connecting to the
                                    host. Defaults to HTTP.
                                  type: string
                              required:
                              - port
                              type: object
                            initialDelaySeconds:
                              description: 'Number of seconds after the container
                                has started before liveness probes are initiated.
                                More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                            periodSeconds:
                              description: How often (in seconds) to perform the probe.
                                Default to 10 seconds. Minimum value is 1.
                              format: int32
                              type: integer
                            successThreshold:
                              description: Minimum consecutive successes for the probe
                                to be considered successful after having failed. Defaults
                                to 1. Must be 1 for liveness and startup. Minimum
                                value is 1.
                              format: int32
                              type: integer
                            tcpSocket:
                              description: 'TCPSocket specifies an action involving
                                a TCP port. TCP hooks not yet supported TODO: implement
                                a realistic TCP lifecycle hook'
                              properties:
                                host:
                                  description: 'Optional: Host name to connect to,
                                    defaults to the pod IP.'
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Number or name of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                              required:
                              - port
                              type: object
                            terminationGracePeriodSeconds:
                              description: Optional duration in seconds the pod needs
                                to terminate gracefully upon probe failure. The grace
                                period is the duration in seconds after the processes
                                running in the pod are sent a termination signal and
                                the time when the processes are forcibly halted with
                                a kill signal. Set this value longer than the expected
                                cleanup time for your process. If this value is nil,
                                the pod's terminationGracePeriodSeconds will be used.
                                Otherwise, this value overrides the value provided
                                by the pod spec. Value must be non-negative integer.
                                The value zero indicates stop immediately via the
                                kill signal (no opportunity to shut down). This is
                                an alpha field and requires enabling ProbeTerminationGracePeriod
                                feature gate.
                              format: int64
                              type: integer
                            timeoutSeconds:
                              description: 'Number of seconds after which the probe
                                times out. Defaults to 1 second. Minimum value is
                                1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                          type: object
                        name:
                          description: Name of the container specified as a DNS_LABEL.
                            Each container in a pod must have a unique name (DNS_LABEL).
                            Cannot be updated.
                          type: string
                        ports:
                          description: List of ports to expose from the container.
                            Exposing a port here gives the system additional information
                            about the network connections a container uses, but is
                            primarily informational. Not specifying a port here DOES
                            NOT prevent that port from being exposed. Any port which
                            is listening on the default "0.0.0.0" address inside a
                            container will be accessible from the network. Cannot
                            be updated.
                          items:
                            description: ContainerPort represents a network port in
                              a single container.
                            properties:
                              containerPort:
                                description: Number of port to expose on the pod's
                                  IP address. This must be a valid port number, 0
                                  < x < 65536.
                                format: int32
                                type: integer
                              hostIP:
                                description: What host IP to bind the external port
                                  to.
                                type: string
                              hostPort:
                                description: Number of port to expose on the host.
                                  If specified, this must be a valid port number,
                                  0 < x < 65536. If HostNetwork is specified, this
                                  must match ContainerPort. Most containers do not
                                  need this.
                                format: int32
                                type: integer
                              name:
                                description: If specified, this must be an IANA_SVC_NAME
                                  and unique within the pod. Each named port in a
                                  pod must have a unique name. Name for the port that
                                  can be referred to by services.
                                type: string
                              protocol:
                                default: TCP
                                description: Protocol for port. Must be UDP, TCP,
                                  or SCTP. Defaults to "TCP".
                                type: string
                            required:
                            - containerPort
                            type: object
                          type: array
                          x-kubernetes-list-map-keys:
                          - containerPort
                          - protocol
                          x-kubernetes-list-type: map
                        readinessProbe:
                          description: 'Periodic probe of container service readiness.
                            Container will be removed from service endpoints if the
                            probe fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                          properties:
                            exec:
                              description: One and only one of the following should
                                be specified. Exec specifies the action to take.
                              properties:
                                command:
                                  description: Command is the command line to execute
                                    inside the container, the working directory for
                                    the command  is root ('/') in the container's
                                    filesystem. The command is simply exec'd, it is
                                    not run inside a shell, so traditional shell instructions
                                    ('|', etc) won't work. To use a shell, you need
                                    to explicitly call out to that shell. Exit status
                                    of 0 is treated as live/healthy and non-zero is
                                    unhealthy.
                                  items:
                                    type: string
                                  type: array
                              type: object
                            failureThreshold:
                              description: Minimum consecutive failures for the probe
                                to be considered failed after having succeeded. Defaults
                                to 3. Minimum value is 1.
                              format: int32
                              type: integer
                            httpGet:
                              description: HTTPGet specifies the http request to perform.
                              properties:
                                host:
                                  description: Host name to connect to, defaults to
                                    the pod IP. You probably want to set "Host" in
                                    httpHeaders instead.
                                  type: string
                                httpHeaders:
                                  description: Custom headers to set in the request.
                                    HTTP allows repeated headers.
                                  items:
                                    description: HTTPHeader describes a custom header
                                      to be used in HTTP probes
                                    properties:
                                      name:
                                        description: The header field name
                                        type: string
                                      value:
                                        description: The header field value
                                        type: string
                                    required:
                                    - name
                                    - value
                                    type: object
                                  type: array
                                path:
                                  description: Path to access on the HTTP server.
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Name or number of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                                scheme:
                                  description: Scheme to use for connecting to the
                                    host. Defaults to HTTP.
                                  type: string
                              required:
                              - port
                              type: object
                            initialDelaySeconds:
                              description: 'Number of seconds after the container
                                has started before liveness probes are initiated.
                                More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                            periodSeconds:
                              description: How often (in seconds) to perform the probe.
                                Default to 10 seconds. Minimum value is 1.
                              format: int32
                              type: integer
                            successThreshold:
                              description: Minimum consecutive successes for the probe
                                to be considered successful after having failed. Defaults
                                to 1. Must be 1 for liveness and startup. Minimum
                                value is 1.
                              format: int32
                              type: integer
                            tcpSocket:
                              description: 'TCPSocket specifies an action involving
                                a TCP port. TCP hooks not yet supported TODO: implement
                                a realistic TCP lifecycle hook'
                              properties:
                                host:
                                  description: 'Optional: Host name to connect to,
                                    defaults to the pod IP.'
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Number or name of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                              required:
                              - port
                              type: object
                            terminationGracePeriodSeconds:
                              description: Optional duration in seconds the pod needs
                                to terminate gracefully upon probe failure. The grace
                                period is the duration in seconds after the processes
                                running in the pod are sent a termination signal and
                                the time when the processes are forcibly halted with
                                a kill signal. Set this value longer than the expected
                                cleanup time for your process. If this value is nil,
                                the pod's terminationGracePeriodSeconds will be used.
                                Otherwise, this value overrides the value provided
                                by the pod spec. Value must be non-negative integer.
                                The value zero indicates stop immediately via the
                                kill signal (no opportunity to shut down). This is
                                an alpha field and requires enabling ProbeTerminationGracePeriod
                                feature gate.
                              format: int64
                              type: integer
                            timeoutSeconds:
                              description: 'Number of seconds after which the probe
                                times out. Defaults to 1 second. Minimum value is
                                1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                          type: object
                        resources:
                          description: 'Compute Resources required by this container.
                            Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                          properties:
                            limits:
                              additionalProperties:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                x-kubernetes-int-or-string: true
                              description: 'Limits describes the maximum amount of
                                compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                              type: object
                            requests:
                              additionalProperties:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                x-kubernetes-int-or-string: true
                              description: 'Requests describes the minimum amount
                                of compute resources required. If Requests is omitted
                                for a container, it defaults to Limits if that is
                                explicitly specified, otherwise to an implementation-defined
                                value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                              type: object
                          type: object
                        securityContext:
                          description: 'Security options the pod should run with.
                            More info: https://kubernetes.io/docs/concepts/policy/security-context/
                            More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/'
                          properties:
                            allowPrivilegeEscalation:
                              description: 'AllowPrivilegeEscalation controls whether
                                a process can gain more privileges than its parent
                                process. This bool directly controls if the no_new_privs
                                flag will be set on the container process. AllowPrivilegeEscalation
                                is true always when the container is: 1) run as Privileged
                                2) has CAP_SYS_ADMIN'
                              type: boolean
                            capabilities:
                              description: The capabilities to add/drop when running
                                containers. Defaults to the default set of capabilities
                                granted by the container runtime.
                              properties:
                                add:
                                  description: Added capabilities
                                  items:
                                    description: Capability represent POSIX capabilities
                                      type
                                    type: string
                                  type: array
                                drop:
                                  description: Removed capabilities
                                  items:
                                    description: Capability represent POSIX capabilities
                                      type
                                    type: string
                                  type: array
                              type: object
                            privileged:
                              description: Run container in privileged mode. Processes
                                in privileged containers are essentially equivalent
                                to root on the host. Defaults to false.
                              type: boolean
                            procMount:
                              description: procMount denotes the type of proc mount
                                to use for the containers. The default is DefaultProcMount
                                which uses the container runtime defaults for readonly
                                paths and masked paths. This requires the ProcMountType
                                feature flag to be enabled.
                              type: string
                            readOnlyRootFilesystem:
                              description: Whether this container has a read-only
                                root filesystem. Default is false.
                              type: boolean
                            runAsGroup:
                              description: The GID to run the entrypoint of the container
                                process. Uses runtime default if unset. May also be
                                set in PodSecurityContext.  If set in both SecurityContext
                                and PodSecurityContext, the value specified in SecurityContext
                                takes precedence.
                              format: int64
                              type: integer
                            runAsNonRoot:
                              description: Indicates that the container must run as
                                a non-root user. If true, the Kubelet will validate
                                the image at runtime to ensure that it does not run
                                as UID 0 (root) and fail to start the container if
                                it does. If unset or false, no such validation will
                                be performed. May also be set in PodSecurityContext.  If
                                set in both SecurityContext and PodSecurityContext,
                                the value specified in SecurityContext takes precedence.
                              type: boolean
                            runAsUser:
                              description: The UID to run the entrypoint of the container
                                process. Defaults to user specified in image metadata
                                if unspecified. May also be set in PodSecurityContext.  If
                                set in both SecurityContext and PodSecurityContext,
                                the value specified in SecurityContext takes precedence.
                              format: int64
                              type: integer
                            seLinuxOptions:
                              description: The SELinux context to be applied to the
                                container. If unspecified, the container runtime will
                                allocate a random SELinux context for each container.  May
                                also be set in PodSecurityContext.  If set in both
                                SecurityContext and PodSecurityContext, the value
                                specified in SecurityContext takes precedence.
                              properties:
                                level:
                                  description: Level is SELinux level label that applies
                                    to the container.
                                  type: string
                                role:
                                  description: Role is a SELinux role label that applies
                                    to the container.
                                  type: string
                                type:
                                  description: Type is a SELinux type label that applies
                                    to the container.
                                  type: string
                                user:
                                  description: User is a SELinux user label that applies
                                    to the container.
                                  type: string
                              type: object
                            seccompProfile:
                              description: The seccomp options to use by this container.
                                If seccomp options are provided at both the pod &
                                container level, the container options override the
                                pod options.
                              properties:
                                localhostProfile:
                                  description: localhostProfile indicates a profile
                                    defined in a file on the node should be used.
                                    The profile must be preconfigured on the node
                                    to work. Must be a descending path, relative to
                                    the kubelet's configured seccomp profile location.
                                    Must only be set if type is "Localhost".
                                  type: string
                                type:
                                  description: "type indicates which kind of seccomp
                                    profile will be applied. Valid options are: \n
                                    Localhost - a profile defined in a file on the
                                    node should be used. RuntimeDefault - the container
                                    runtime default profile should be used. Unconfined
                                    - no profile should be applied."
                                  type: string
                              required:
                              - type
                              type: object
                            windowsOptions:
                              description: The Windows specific settings applied to
                                all containers. If unspecified, the options from the
                                PodSecurityContext will be used. If set in both SecurityContext
                                and PodSecurityContext, the value specified in SecurityContext
                                takes precedence.
                              properties:
                                gmsaCredentialSpec:
                                  description: GMSACredentialSpec is where the GMSA
                                    admission webhook (https://github.com/kubernetes-sigs/windows-gmsa)
                                    inlines the contents of the GMSA credential spec
                                    named by the GMSACredentialSpecName field.
                                  type: string
                                gmsaCredentialSpecName:
                                  description: GMSACredentialSpecName is the name
                                    of the GMSA credential spec to use.
                                  type: string
                                runAsUserName:
                                  description: The UserName in Windows to run the
                                    entrypoint of the container process. Defaults
                                    to the user specified in image metadata if unspecified.
                                    May also be set in PodSecurityContext. If set
                                    in both SecurityContext and PodSecurityContext,
                                    the value specified in SecurityContext takes precedence.
                                  type: string
                              type: object
                          type: object
                        startupProbe:
                          description: 'StartupProbe indicates that the Pod has successfully
                            initialized. If specified, no other probes are executed
                            until this completes successfully. If this probe fails,
                            the Pod will be restarted, just as if the livenessProbe
                            failed. This can be used to provide different probe parameters
                            at the beginning of a Pod''s lifecycle, when it might
                            take a long time to load data or warm a cache, than during
                            steady-state operation. This cannot be updated. More info:
                            https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                          properties:
                            exec:
                              description: One and only one of the following should
                                be specified. Exec specifies the action to take.
                              properties:
                                command:
                                  description: Command is the command line to execute
                                    inside the container, the working directory for
                                    the command  is root ('/') in the container's
                                    filesystem. The command is simply exec'd, it is
                                    not run inside a shell, so traditional shell instructions
                                    ('|', etc) won't work. To use a shell, you need
                                    to explicitly call out to that shell. Exit status
                                    of 0 is treated as live/healthy and non-zero is
                                    unhealthy.
                                  items:
                                    type: string
                                  type: array
                              type: object
                            failureThreshold:
                              description: Minimum consecutive failures for the probe
                                to be considered failed after having succeeded. Defaults
                                to 3. Minimum value is 1.
                              format: int32
                              type: integer
                            httpGet:
                              description: HTTPGet specifies the http request to perform.
                              properties:
                                host:
                                  description: Host name to connect to, defaults to
                                    the pod IP. You probably want to set "Host" in
                                    httpHeaders instead.
                                  type: string
                                httpHeaders:
                                  description: Custom headers to set in the request.
                                    HTTP allows repeated headers.
                                  items:
                                    description: HTTPHeader describes a custom header
                                      to be used in HTTP probes
                                    properties:
                                      name:
                                        description: The header field name
                                        type: string
                                      value:
                                        description: The header field value
                                        type: string
                                    required:
                                    - name
                                    - value
                                    type: object
                                  type: array
                                path:
                                  description: Path to access on the HTTP server.
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Name or number of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                                scheme:
                                  description: Scheme to use for connecting to the
                                    host. Defaults to HTTP.
                                  type: string
                              required:
                              - port
                              type: object
                            initialDelaySeconds:
                              description: 'Number of seconds after the container
                                has started before liveness probes are initiated.
                                More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                            periodSeconds:
                              description: How often (in seconds) to perform the probe.
                                Default to 10 seconds. Minimum value is 1.
                              format: int32
                              type: integer
                            successThreshold:
                              description: Minimum consecutive successes for the probe
                                to be considered successful after having failed. Defaults
                                to 1. Must be 1 for liveness and startup. Minimum
                                value is 1.
                              format: int32
                              type: integer
                            tcpSocket:
                              description: 'TCPSocket specifies an action involving
                                a TCP port. TCP hooks not yet supported TODO: implement
                                a realistic TCP lifecycle hook'
                              properties:
                                host:
                                  description: 'Optional: Host name to connect to,
                                    defaults to the pod IP.'
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Number or name of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                              required:
                              - port
                              type: object
                            terminationGracePeriodSeconds:
                              description: Optional duration in seconds the pod needs
                                to terminate gracefully upon probe failure. The grace
                                period is the duration in seconds after the processes
                                running in the pod are sent a termination signal and
                                the time when the processes are forcibly halted with
                                a kill signal. Set this value longer than the expected
                                cleanup time for your process. If this value is nil,
                                the pod's terminationGracePeriodSeconds will be used.
                                Otherwise, this value overrides the value provided
                                by the pod spec. Value must be non-negative integer.
                                The value zero indicates stop immediately via the
                                kill signal (no opportunity to shut down). This is
                                an alpha field and requires enabling ProbeTerminationGracePeriod
                                feature gate.
                              format: int64
                              type: integer
                            timeoutSeconds:
                              description: 'Number of seconds after which the probe
                                times out. Defaults to 1 second. Minimum value is
                                1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                          type: object
                        stdin:
                          description: Whether this container should allocate a buffer
                            for stdin in the container runtime. If this is not set,
                            reads from stdin in the container will always result in
                            EOF. Default is false.
                          type: boolean
                        stdinOnce:
                          description: Whether the container runtime should close
                            the stdin channel after it has been opened by a single
                            attach. When stdin is true the stdin stream will remain
                            open across multiple attach sessions. If stdinOnce is
                            set to true, stdin is opened on container start, is empty
                            until the first client attaches to stdin, and then remains
                            open and accepts data until the client disconnects, at
                            which time stdin is closed and remains closed until the
                            container is restarted. If this flag is false, a container
                            processes that reads from stdin will never receive an
                            EOF. Default is false
                          type: boolean
                        terminationMessagePath:
                          description: 'Optional: Path at which the file to which
                            the container''s termination message will be written is
                            mounted into the container''s filesystem. Message written
                            is intended to be brief final status, such as an assertion
                            failure message. Will be truncated by the node if greater
                            than 4096 bytes. The total message length across all containers
                            will be limited to 12kb. Defaults to /dev/termination-log.
                            Cannot be updated.'
                          type: string
                        terminationMessagePolicy:
                          description: Indicate how the termination message should
                            be populated. File will use the contents of terminationMessagePath
                            to populate the container status message on both success
                            and failure. FallbackToLogsOnError will use the last chunk
                            of container log output if the termination message file
                            is empty and the container exited with an error. The log
                            output is limited to 2048 bytes or 80 lines, whichever
                            is smaller. Defaults to File. Cannot be updated.
                          type: string
                        tty:
                          description: Whether this container should allocate a TTY
                            for itself, also requires 'stdin' to be true. Default
                            is false.
                          type: boolean
                        volumeDevices:
                          description: volumeDevices is the list of block devices
                            to be used by the container.
                          items:
                            description: volumeDevice describes a mapping of a raw
                              block device within a container.
                            properties:
                              devicePath:
                                description: devicePath is the path inside of the
                                  container that the device will be mapped to.
                                type: string
                              name:
                                description: name must match the name of a persistentVolumeClaim
                                  in the pod
                                type: string
                            required:
                            - devicePath
                            - name
                            type: object
                          type: array
                        volumeMounts:
                          description: Pod volumes to mount into the container's filesystem.
                            Cannot be updated.
                          items:
                            description: VolumeMount describes a mounting of a Volume
                              within a container.
                            properties:
                              mountPath:
                                description: Path within the container at which the
                                  volume should be mounted.  Must not contain ':'.
                                type: string
                              mountPropagation:
                                description: mountPropagation determines how mounts
                                  are propagated from the host to container and the
                                  other way around. When not set, MountPropagationNone
                                  is used. This field is beta in 1.10.
                                type: string
                              name:
                                description: This must match the Name of a Volume.
                                type: string
                              readOnly:
                                description: Mounted read-only if true, read-write
                                  otherwise (false or unspecified). Defaults to false.
                                type: boolean
                              subPath:
                                description: Path within the volume from which the
                                  container's volume should be mounted. Defaults to
                                  "" (volume's root).
                                type: string
                              subPathExpr:
                                description: Expanded path within the volume from
                                  which the container's volume should be mounted.
                                  Behaves similarly to SubPath but environment variable
                                  references $(VAR_NAME) are expanded using the container's
                                  environment. Defaults to "" (volume's root). SubPathExpr
                                  and SubPath are mutually exclusive.
                                type: string
                            required:
                            - mountPath
                            - name
                            type: object
                          type: array
                        workingDir:
                          description: Container's working directory. If not specified,
                            the container runtime's default will be used, which might
                            be configured in the container image. Cannot be updated.
                          type: string
                      required:
                      - name
                      type: object
                    type: array
                  dnsConfig:
                    description: Specifies the DNS parameters of a pod. Parameters
                      specified here will be merged to the generated DNS configuration
                      based on DNSPolicy.
                    properties:
                      nameservers:
                        description: A list of DNS name server IP addresses. This
                          will be appended to the base nameservers generated from
                          DNSPolicy. Duplicated nameservers will be removed.
                        items:
                          type: string
                        type: array
                      options:
                        description: A list of DNS resolver options. This will be
                          merged with the base options generated from DNSPolicy. Duplicated
                          entries will be removed. Resolution options given in Options
                          will override those that appear in the base DNSPolicy.
                        items:
                          description: PodDNSConfigOption defines DNS resolver options
                            of a pod.
                          properties:
                            name:
                              description: Required.
                              type: string
                            value:
                              type: string
                          type: object
                        type: array
                      searches:
                        description: A list of DNS search domains for host-name lookup.
                          This will be appended to the base search paths generated
                          from DNSPolicy. Duplicated search paths will be removed.
                        items:
                          type: string
                        type: array
                    type: object
                  dnsPolicy:
                    description: Set DNS policy for the pod. Defaults to "ClusterFirst".
                      Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst',
                      'Default' or 'None'. DNS parameters given in DNSConfig will
                      be merged with the policy selected with DNSPolicy. To have DNS
                      options set along with hostNetwork, you have to specify DNS
                      policy explicitly to 'ClusterFirstWithHostNet'.
                    type: string
                  enableServiceLinks:
                    description: 'EnableServiceLinks indicates whether information
                      about services should be injected into pod''s environment variables,
                      matching the syntax of Docker links. Optional: Defaults to true.'
                    type: boolean
                  ephemeralContainers:
                    description: List of ephemeral containers run in this pod. Ephemeral
                      containers may be run in an existing pod to perform user-initiated
                      actions such as debugging. This list cannot be specified when
                      creating a pod, and it cannot be modified by updating the pod
                      spec. In order to add an ephemeral container to an existing
                      pod, use the pod's ephemeralcontainers subresource. This field
                      is alpha-level and is only honored by servers that enable the
                      EphemeralContainers feature.
                    items:
                      description: An EphemeralContainer is a container that may be
                        added temporarily to an existing pod for user-initiated activities
                        such as debugging. Ephemeral containers have no resource or
                        scheduling guarantees, and they will not be restarted when
                        they exit or when a pod is removed or restarted. If an ephemeral
                        container causes a pod to exceed its resource allocation,
                        the pod may be evicted. Ephemeral containers may not be added
                        by directly updating the pod spec. They must be added via
                        the pod's ephemeralcontainers subresource, and they will appear
                        in the pod spec once added. This is an alpha feature enabled
                        by the EphemeralContainers feature flag.
                      properties:
                        args:
                          description: 'Arguments to the entrypoint. The docker image''s
                            CMD is used if this is not provided. Variable references
                            $(VAR_NAME) are expanded using the container''s environment.
                            If a variable cannot be resolved, the reference in the
                            input string will be unchanged. The $(VAR_NAME) syntax
                            can be escaped with a double $$, ie: $$(VAR_NAME). Escaped
                            references will never be expanded, regardless of whether
                            the variable exists or not. Cannot be updated. More info:
                            https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
                          items:
                            type: string
                          type: array
                        command:
                          description: 'Entrypoint array. Not executed within a shell.
                            The docker image''s ENTRYPOINT is used if this is not
                            provided. Variable references $(VAR_NAME) are expanded
                            using the container''s environment. If a variable cannot
                            be resolved, the reference in the input string will be
                            unchanged. The $(VAR_NAME) syntax can be escaped with
                            a double $$, ie: $$(VAR_NAME). Escaped references will
                            never be expanded, regardless of whether the variable
                            exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell'
                          items:
                            type: string
                          type: array
                        env:
                          description: List of environment variables to set in the
                            container. Cannot be updated.
                          items:
                            description: EnvVar represents an environment variable
                              present in a Container.
                            properties:
                              name:
                                description: Name of the environment variable. Must
                                  be a C_IDENTIFIER.
                                type: string
                              value:
                                description: 'Variable references $(VAR_NAME) are
                                  expanded using the previous defined environment
                                  variables in the container and any service environment
                                  variables. If a variable cannot be resolved, the
                                  reference in the input string will be unchanged.
                                  The $(VAR_NAME) syntax can be escaped with a double
                                  $$, ie: $$(VAR_NAME). Escaped references will never
                                  be expanded, regardless of whether the variable
                                  exists or not. Defaults to "".'
                                type: string
                              valueFrom:
                                description: Source for the environment variable's
                                  value. Cannot be used if value is not empty.
                                properties:
                                  configMapKeyRef:
                                    description: Selects a key of a ConfigMap.
                                    properties:
                                      key:
                                        description: The key to select.
                                        type: string
                                      name:
                                        description: 'Name of the referent. More info:
                                          https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                                          TODO: Add other useful fields. apiVersion,
                                          kind, uid?'
                                        type: string
                                      optional:
                                        description: Specify whether the ConfigMap
                                          or its key must be defined
                                        type: boolean
                                    required:
                                    - key
                                    type: object
                                  fieldRef:
                                    description: 'Selects a field of the pod: supports
                                      metadata.name, metadata.namespace, `metadata.labels[''<KEY>'']`,
                                      `metadata.annotations[''<KEY>'']`, spec.nodeName,
                                      spec.serviceAccountName, status.hostIP, status.podIP,
                                      status.podIPs.'
                                    properties:
                                      apiVersion:
                                        description: Version of the schema the FieldPath
                                          is written in terms of, defaults to "v1".
                                        type: string
                                      fieldPath:
                                        description: Path of the field to select in
                                          the specified API version.
                                        type: string
                                    required:
                                    - fieldPath
                                    type: object
                                  resourceFieldRef:
                                    description: 'Selects a resource of the container:
                                      only resources limits and requests (limits.cpu,
                                      limits.memory, limits.ephemeral-storage, requests.cpu,
                                      requests.memory and requests.ephemeral-storage)
                                      are currently supported.'
                                    properties:
                                      containerName:
                                        description: 'Container name: required for
                                          volumes, optional for env vars'
                                        type: string
                                      divisor:
                                        anyOf:
                                        - type: integer
                                        - type: string
                                        description: Specifies the output format of
                                          the exposed resources, defaults to "1"
                                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                        x-kubernetes-int-or-string: true
                                      resource:
                                        description: 'Required: resource to select'
                                        type: string
                                    required:
                                    - resource
                                    type: object
                                  secretKeyRef:
                                    description: Selects a key of a secret in the
                                      pod's namespace
                                    properties:
                                      key:
                                        description: The key of the secret to select
                                          from.  Must be a valid secret key.
                                        type: string
                                      name:
                                        description: 'Name of the referent. More info:
                                          https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                                          TODO: Add other useful fields. apiVersion,
                                          kind, uid?'
                                        type: string
                                      optional:
                                        description: Specify whether the Secret or
                                          its key must be defined
                                        type: boolean
                                    required:
                                    - key
                                    type: object
                                type: object
                            required:
                            - name
                            type: object
                          type: array
                        envFrom:
                          description: List of sources to populate environment variables
                            in the container. The keys defined within a source must
                            be a C_IDENTIFIER. All invalid keys will be reported as
                            an event when the container is starting. When a key exists
                            in multiple sources, the value associated with the last
                            source will take precedence. Values defined by an Env
                            with a duplicate key will take precedence. Cannot be updated.
                          items:
                            description: EnvFromSource represents the source of a
                              set of ConfigMaps
                            properties:
                              configMapRef:
                                description: The ConfigMap to select from
                                properties:
                                  name:
                                    description: 'Name of the referent. More info:
                                      https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                                      TODO: Add other useful fields. apiVersion, kind,
                                      uid?'
                                    type: string
                                  optional:
                                    description: Specify whether the ConfigMap must
                                      be defined
                                    type: boolean
                                type: object
                              prefix:
                                description: An optional identifier to prepend to
                                  each key in the ConfigMap. Must be a C_IDENTIFIER.
                                type: string
                              secretRef:
                                description: The Secret to select from
                                properties:
                                  name:
                                    description: 'Name of the referent. More info:
                                      https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                                      TODO: Add other useful fields. apiVersion, kind,
                                      uid?'
                                    type: string
                                  optional:
                                    description: Specify whether the Secret must be
                                      defined
                                    type: boolean
                                type: object
                            type: object
                          type: array
                        image:
                          description: 'Docker image name. More info: https://kubernetes.io/docs/concepts/containers/images'
                          type: string
                        imagePullPolicy:
                          description: 'Image pull policy. One of Always, Never, IfNotPresent.
                            Defaults to Always if :latest tag is specified, or IfNotPresent
                            otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images'
                          type: string
                        lifecycle:
                          description: Lifecycle is not allowed for ephemeral containers.
                          properties:
                            postStart:
                              description: 'PostStart is called immediately after
                                a container is created. If the handler fails, the
                                container is terminated and restarted according to
                                its restart policy. Other management of the container
                                blocks until the hook completes. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
                              properties:
                                exec:
                                  description: One and only one of the following should
                                    be specified. Exec specifies the action to take.
                                  properties:
                                    command:
                                      description: Command is the command line to
                                        execute inside the container, the working
                                        directory for the command  is root ('/') in
                                        the container's filesystem. The command is
                                        simply exec'd, it is not run inside a shell,
                                        so traditional shell instructions ('|', etc)
                                        won't work. To use a shell, you need to explicitly
                                        call out to that shell. Exit status of 0 is
                                        treated as live/healthy and non-zero is unhealthy.
                                      items:
                                        type: string
                                      type: array
                                  type: object
                                httpGet:
                                  description: HTTPGet specifies the http request
                                    to perform.
                                  properties:
                                    host:
                                      description: Host name to connect to, defaults
                                        to the pod IP. You probably want to set "Host"
                                        in httpHeaders instead.
                                      type: string
                                    httpHeaders:
                                      description: Custom headers to set in the request.
                                        HTTP allows repeated headers.
                                      items:
                                        description: HTTPHeader describes a custom
                                          header to be used in HTTP probes
                                        properties:
                                          name:
                                            description: The header field name
                                            type: string
                                          value:
                                            description: The header field value
                                            type: string
                                        required:
                                        - name
                                        - value
                                        type: object
                                      type: array
                                    path:
                                      description: Path to access on the HTTP server.
                                      type: string
                                    port:
                                      anyOf:
                                      - type: integer
                                      - type: string
                                      description: Name or number of the port to access
                                        on the container. Number must be in the range
                                        1 to 65535. Name must be an IANA_SVC_NAME.
                                      x-kubernetes-int-or-string: true
                                    scheme:
                                      description: Scheme to use for connecting to
                                        the host. Defaults to HTTP.
                                      type: string
                                  required:
                                  - port
                                  type: object
                                tcpSocket:
                                  description: 'TCPSocket specifies an action involving
                                    a TCP port. TCP hooks not yet supported TODO:
                                    implement a realistic TCP lifecycle hook'
                                  properties:
                                    host:
                                      description: 'Optional: Host name to connect
                                        to, defaults to the pod IP.'
                                      type: string
                                    port:
                                      anyOf:
                                      - type: integer
                                      - type: string
                                      description: Number or name of the port to access
                                        on the container. Number must be in the range
                                        1 to 65535. Name must be an IANA_SVC_NAME.
                                      x-kubernetes-int-or-string: true
                                  required:
                                  - port
                                  type: object
                              type: object
                            preStop:
                              description: 'PreStop is called immediately before a
                                container is terminated due to an API request or management
                                event such as liveness/startup probe failure, preemption,
                                resource contention, etc. The handler is not called
                                if the container crashes or exits. The reason for
                                termination is passed to the handler. The Pod''s termination
                                grace period countdown begins before the PreStop hooked
                                is executed. Regardless of the outcome of the handler,
                                the container will eventually terminate within the
                                Pod''s termination grace period. Other management
                                of the container blocks until the hook completes or
                                until the termination grace period is reached. More
                                info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks'
                              properties:
                                exec:
                                  description: One and only one of the following should
                                    be specified. Exec specifies the action to take.
                                  properties:
                                    command:
                                      description: Command is the command line to
                                        execute inside the container, the working
                                        directory for the command  is root ('/') in
                                        the container's filesystem. The command is
                                        simply exec'd, it is not run inside a shell,
                                        so traditional shell instructions ('|', etc)
                                        won't work. To use a shell, you need to explicitly
                                        call out to that shell. Exit status of 0 is
                                        treated as live/healthy and non-zero is unhealthy.
                                      items:
                                        type: string
                                      type: array
                                  type: object
                                httpGet:
                                  description: HTTPGet specifies the http request
                                    to perform.
                                  properties:
                                    host:
                                      description: Host name to connect to, defaults
                                        to the pod IP. You probably want to set "Host"
                                        in httpHeaders instead.
                                      type: string
                                    httpHeaders:
                                      description: Custom headers to set in the request.
                                        HTTP allows repeated headers.
                                      items:
                                        description: HTTPHeader describes a custom
                                          header to be used in HTTP probes
                                        properties:
                                          name:
                                            description: The header field name
                                            type: string
                                          value:
                                            description: The header field value
                                            type: string
                                        required:
                                        - name
                                        - value
                                        type: object
                                      type: array
                                    path:
                                      description: Path to access on the HTTP server.
                                      type: string
                                    port:
                                      anyOf:
                                      - type: integer
                                      - type: string
                                      description: Name or number of the port to access
                                        on the container. Number must be in the range
                                        1 to 65535. Name must be an IANA_SVC_NAME.
                                      x-kubernetes-int-or-string: true
                                    scheme:
                                      description: Scheme to use for connecting to
                                        the host. Defaults to HTTP.
                                      type: string
                                  required:
                                  - port
                                  type: object
                                tcpSocket:
                                  description: 'TCPSocket specifies an action involving
                                    a TCP port. TCP hooks not yet supported TODO:
                                    implement a realistic TCP lifecycle hook'
                                  properties:
                                    host:
                                      description: 'Optional: Host name to connect
                                        to, defaults to the pod IP.'
                                      type: string
                                    port:
                                      anyOf:
                                      - type: integer
                                      - type: string
                                      description: Number or name of the port to access
                                        on the container. Number must be in the range
                                        1 to 65535. Name must be an IANA_SVC_NAME.
                                      x-kubernetes-int-or-string: true
                                  required:
                                  - port
                                  type: object
                              type: object
                          type: object
                        livenessProbe:
                          description: Probes are not allowed for ephemeral containers.
                          properties:
                            exec:
                              description: One and only one of the following should
                                be specified. Exec specifies the action to take.
                              properties:
                                command:
                                  description: Command is the command line to execute
                                    inside the container, the working directory for
                                    the command  is root ('/') in the container's
                                    filesystem. The command is simply exec'd, it is
                                    not run inside a shell, so traditional shell instructions
                                    ('|', etc) won't work. To use a shell, you need
                                    to explicitly call out to that shell. Exit status
                                    of 0 is treated as live/healthy and non-zero is
                                    unhealthy.
                                  items:
                                    type: string
                                  type: array
                              type: object
                            failureThreshold:
                              description: Minimum consecutive failures for the probe
                                to be considered failed after having succeeded. Defaults
                                to 3. Minimum value is 1.
                              format: int32
                              type: integer
                            httpGet:
                              description: HTTPGet specifies the http request to perform.
                              properties:
                                host:
                                  description: Host name to connect to, defaults to
                                    the pod IP. You probably want to set "Host" in
                                    httpHeaders instead.
                                  type: string
                                httpHeaders:
                                  description: Custom headers to set in the request.
                                    HTTP allows repeated headers.
                                  items:
                                    description: HTTPHeader describes a custom header
                                      to be used in HTTP probes
                                    properties:
                                      name:
                                        description: The header field name
                                        type: string
                                      value:
                                        description: The header field value
                                        type: string
                                    required:
                                    - name
                                    - value
                                    type: object
                                  type: array
                                path:
                                  description: Path to access on the HTTP server.
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Name or number of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                                scheme:
                                  description: Scheme to use for connecting to the
                                    host. Defaults to HTTP.
                                  type: string
                              required:
                              - port
                              type: object
                            initialDelaySeconds:
                              description: 'Number of seconds after the container
                                has started before liveness probes are initiated.
                                More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                            periodSeconds:
                              description: How often (in seconds) to perform the probe.
                                Default to 10 seconds. Minimum value is 1.
                              format: int32
                              type: integer
                            successThreshold:
                              description: Minimum consecutive successes for the probe
                                to be considered successful after having failed. Defaults
                                to 1. Must be 1 for liveness and startup. Minimum
                                value is 1.
                              format: int32
                              type: integer
                            tcpSocket:
                              description: 'TCPSocket specifies an action involving
                                a TCP port. TCP hooks not yet supported TODO: implement
                                a realistic TCP lifecycle hook'
                              properties:
                                host:
                                  description: 'Optional: Host name to connect to,
                                    defaults to the pod IP.'
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Number or name of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                              required:
                              - port
                              type: object
                            terminationGracePeriodSeconds:
                              description: Optional duration in seconds the pod needs
                                to terminate gracefully upon probe failure. The grace
                                period is the duration in seconds after the processes
                                running in the pod are sent a termination signal and
                                the time when the processes are forcibly halted with
                                a kill signal. Set this value longer than the expected
                                cleanup time for your process. If this value is nil,
                                the pod's terminationGracePeriodSeconds will be used.
                                Otherwise, this value overrides the value provided
                                by the pod spec. Value must be non-negative integer.
                                The value zero indicates stop immediately via the
                                kill signal (no opportunity to shut down). This is
                                an alpha field and requires enabling ProbeTerminationGracePeriod
                                feature gate.
                              format: int64
                              type: integer
                            timeoutSeconds:
                              description: 'Number of seconds after which the probe
                                times out. Defaults to 1 second. Minimum value is
                                1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                          type: object
                        name:
                          description: Name of the ephemeral container specified as
                            a DNS_LABEL. This name must be unique among all containers,
                            init containers and ephemeral containers.
                          type: string
                        ports:
                          description: Ports are not allowed for ephemeral containers.
                          items:
                            description: ContainerPort represents a network port in
                              a single container.
                            properties:
                              containerPort:
                                description: Number of port to expose on the pod's
                                  IP address. This must be a valid port number, 0
                                  < x < 65536.
                                format: int32
                                type: integer
                              hostIP:
                                description: What host IP to bind the external port
                                  to.
                                type: string
                              hostPort:
                                description: Number of port to expose on the host.
                                  If specified, this must be a valid port number,
                                  0 < x < 65536. If HostNetwork is specified, this
                                  must match ContainerPort. Most containers do not
                                  need this.
                                format: int32
                                type: integer
                              name:
                                description: If specified, this must be an IANA_SVC_NAME
                                  and unique within the pod. Each named port in a
                                  pod must have a unique name. Name for the port that
                                  can be referred to by services.
                                type: string
                              protocol:
                                default: TCP
                                description: Protocol for port. Must be UDP, TCP,
                                  or SCTP. Defaults to "TCP".
                                type: string
                            required:
                            - containerPort
                            type: object
                          type: array
                        readinessProbe:
                          description: Probes are not allowed for ephemeral containers.
                          properties:
                            exec:
                              description: One and only one of the following should
                                be specified. Exec specifies the action to take.
                              properties:
                                command:
                                  description: Command is the command line to execute
                                    inside the container, the working directory for
                                    the command  is root ('/') in the container's
                                    filesystem. The command is simply exec'd, it is
                                    not run inside a shell, so traditional shell instructions
                                    ('|', etc) won't work. To use a shell, you need
                                    to explicitly call out to that shell. Exit status
                                    of 0 is treated as live/healthy and non-zero is
                                    unhealthy.
                                  items:
                                    type: string
                                  type: array
                              type: object
                            failureThreshold:
                              description: Minimum consecutive failures for the probe
                                to be considered failed after having succeeded. Defaults
                                to 3. Minimum value is 1.
                              format: int32
                              type: integer
                            httpGet:
                              description: HTTPGet specifies the http request to perform.
                              properties:
                                host:
                                  description: Host name to connect to, defaults to
                                    the pod IP. You probably want to set "Host" in
                                    httpHeaders instead.
                                  type: string
                                httpHeaders:
                                  description: Custom headers to set in the request.
                                    HTTP allows repeated headers.
                                  items:
                                    description: HTTPHeader describes a custom header
                                      to be used in HTTP probes
                                    properties:
                                      name:
                                        description: The header field name
                                        type: string
                                      value:
                                        description: The header field value
                                        type: string
                                    required:
                                    - name
                                    - value
                                    type: object
                                  type: array
                                path:
                                  description: Path to access on the HTTP server.
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Name or number of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                                scheme:
                                  description: Scheme to use for connecting to the
                                    host. Defaults to HTTP.
                                  type: string
                              required:
                              - port
                              type: object
                            initialDelaySeconds:
                              description: 'Number of seconds after the container
                                has started before liveness probes are initiated.
                                More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                            periodSeconds:
                              description: How often (in seconds) to perform the probe.
                                Default to 10 seconds. Minimum value is 1.
                              format: int32
                              type: integer
                            successThreshold:
                              description: Minimum consecutive successes for the probe
                                to be considered successful after having failed. Defaults
                                to 1. Must be 1 for liveness and startup. Minimum
                                value is 1.
                              format: int32
                              type: integer
                            tcpSocket:
                              description: 'TCPSocket specifies an action involving
                                a TCP port. TCP hooks not yet supported TODO: implement
                                a realistic TCP lifecycle hook'
                              properties:
                                host:
                                  description: 'Optional: Host name to connect to,
                                    defaults to the pod IP.'
                                  type: string
                                port:
                                  anyOf:
                                  - type: integer
                                  - type: string
                                  description: Number or name of the port to access
                                    on the container. Number must be in the range
                                    1 to 65535. Name must be an IANA_SVC_NAME.
                                  x-kubernetes-int-or-string: true
                              required:
                              - port
                              type: object
                            terminationGracePeriodSeconds:
                              description: Optional duration in seconds the pod needs
                                to terminate gracefully upon probe failure. The grace
                                period is the duration in seconds after the processes
                                running in the pod are sent a termination signal and
                                the time when the processes are forcibly halted with
                                a kill signal. Set this value longer than the expected
                                cleanup time for your process. If this value is nil,
                                the pod's terminationGracePeriodSeconds will be used.
                                Otherwise, this value overrides the value provided
                                by the pod spec. Value must be non-negative integer.
                                The value zero indicates stop immediately via the
                                kill signal (no opportunity to shut down). This is
                                an alpha field and requires enabling ProbeTerminationGracePeriod
                                feature gate.
                              format: int64
                              type: integer
                            timeoutSeconds:
                              description: 'Number of seconds after which the probe
                                times out. Defaults to 1 second. Minimum value is
                                1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes'
                              format: int32
                              type: integer
                          type: object
                        resources:
                          description: Resources are not allowed for ephemeral containers.
                            Ephemeral containers use spare resources already allocated
                            to the pod.
                          properties:
                            limits:
                              additionalProperties:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                x-kubernetes-int-or-string: true
                              description: 'Limits describes the maximum amount of
                                compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                              type: object
                            requests:
                              additionalProperties:
                                anyOf:
                                - type: integer
                                - type: string
                                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                x-kubernetes-int-or-string: true
                              description: 'Requests describes the minimum amount
                                of compute resources required. If Requests is omitted
                                for a container, it defaults to Limits if that is
                                explicitly specified, otherwise to an implementation-defined
                                value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
                              type: object
                          type: object
                        securityContext:
                          description: SecurityContext is not allowed for ephemeral
                            containers.
                          properties:
                            allowPrivilegeEscalation:
                              description: 'AllowPrivilegeEscalation controls whether
                                a process can gain more privileges than its parent
                                process. This bool directly controls if the no_new_privs
                                flag will be set on the container process. AllowPrivilegeEscalation
                                is true always when the container is: 1) run as Privileged
                                2) has CAP_SYS_ADMIN'
                              type: boolean
                            capabilities:
                              description: The capabilities to add/drop when running
                                containers. Defaults to the default set of capabilities
                                granted by the container runtime.
                              properties:
                                add:
                                  description: Added capabilities
                                  items:
                                    description: Capability represent POSIX capabilities
                                      type
                                    type: string
                                  type: array
                                drop:
                                  description: Removed capabilities
                                  items:
                                    description: Capability represent POSIX capabilities
                                      type
                                    type: string
                                  type: array
                              type: object
                            privileged:
                              description: Run container in privileged mode. Processes
                                in privileged containers are essentially equivalent
                                to root on the host. Defaults to false.
                              type: boolean
                            procMount:
                              description: procMount denotes the type of proc mount
                                to use for the containers. The default is DefaultProcMount
                                which uses the container runtime defaults for readonly
                                paths and masked paths. This requires the ProcMountType
                                feature flag to be enabled.
                              type: string
                            readOnlyRootFilesystem:
                              description: Whether this container has a read-only
                                root filesystem. Default is false.
                              type: boolean
                            runAsGroup:
                              description: The GID to run the entrypoint of the container
                                process. Uses runtime default if unset. May also be
                                set in PodSecurityContext.  If set in both SecurityContext
                                and PodSecurityContext, the value specified in SecurityContext
                                takes precedence.
                              format: int64
                              type: integer
                            runAsNonRoot:
                              description: Indicates that the container must run as
                                a non-root user. If true, the Kubelet will validate
                                the image at runtime to ensure that it does not run
                                as UID 0 (root) and fail to start the container if
                                it does. If unset or false, no such validation will
                                be performed. May also be set in PodSecurityContext.  If
                                set in both SecurityContext and PodSecurityContext,
                                the value specified in SecurityContext takes precedence.
                              type: boolean
                            runAsUser:
                              description: The UID to run the entrypoint of the container
                                process. Defaults to user specified in image metadata
                                if unspecified. May also be set in PodSecurityContext.  If
                                set in both SecurityContext and PodSecurityContext,
                                the value specified in SecurityContext takes precedence.
                              format: int64
                              type: integer
                            seLinuxOptions:
                              description: The SELinux context to be applied to the
                                container. If unspecified, the container runtime will
                                allocate a random SELinux context for each container.  May
                                also be set in PodSecurityContext.  If set in both
                                SecurityContext and PodSecurityContext, the value
                                specified in SecurityContext takes precedence.
                              properties:
                                level:
                                  description: Level is SELinux level label that applies
                                    to the container.
                                  type: string
                                role:
                                  description: Role is a SELinux role label that applies
                                    to the container.
                                  type: string
                                type:
                                  description: Type is a SELinux type label that applies
                                    to the container.
                                  type: string
                                user:
                                  description: User is a SELinux user label that applies
                                    to the container.
                                  type: string
                              type: object
                            seccompProfile:
                              description: The seccomp options to use by this container.
                                If seccomp options are provided at both the pod &
                                container level, the container options override the
                                pod options.
                              properties:
                                localhostProfile:
                                  description: localhostProfile indicates a profile
                                    defined in a file on the node should be used.
                                    The profile must be preconfigured on the node
                                    to work. Must be a descending path, relative to
                                    the kubelet's configured seccomp profile location.
                                    Must only be set if type is "Localhost".
                                  type: string
                                type:
                                  description: "type indicates which kind of seccomp
                                    profile will be applied. Valid options are: \n
                                    Localhost - a profile defined in a file on the
                                    node should be used. RuntimeDefault - the container
                                    runtime default profile should be used. Unconfined
                                    - no profile should be applied."
                                  type: string
                              required:
                              - type
                              type: object
                            windowsOptions:
                              description: The Windows specific settings applied to
                                all containers. If unspecified, the options from the
                                PodSecurityContext will be used. If set in both SecurityContext
                                and PodSecurityContext, the value specified in SecurityContext
                                takes precedence.
                              properties:
                                gmsaCredentialSpec:
                                  description: GMSACredentialSpec is where the GMSA
                                    admission webhook (https://github.com/kubernetes-sigs/windows-gmsa)
                                    inlines the contents of the GMSA credential spec
                                    named by the GMSACredentialSpecName field.
                                  type: string
                                gmsaCredentialSpecName:

(2)使用Prometheus,但没使用Prometheus Operator(内容太多,只展示名字)

kuberhealthy-prometheus.yaml

(3)使用Prometheus Operator(内容太多,只展示名字)

kuberhealthy-prometheus-operator.yaml

(2)使用Helm安装

# 步骤一 在Kubernetes集群/上下文中创建命名空间“kuberhealthy”:
kubectl create namespace kuberhealthy
# 步骤二 将kuberhealthy repo添加到Helm
helm repo add kuberhealthy https://kuberhealthy.github.io/kuberhealthy/helm-repos

步骤三:根据你的Prometheus实现,使用适当的命令为你的集群安装kuberhealthy:

(1)如果有Prometheus,而没使用Prometheus操作器

helm install --set prometheus.enabled=true -n kuberhealthy kuberhealthy kuberhealthy/kuberhealthy

(2)如果使用Prometheus操作器

helm install --set prometheus.enabled=true --set prometheus.serviceMonitor.enabled=true -n kuberhealthy kuberhealthy kuberhealthy/kuberhealthy

使用步骤

3.1 查看Kuberhealthy pod

使用下面的命令查看Kuberhealthy pod

kubectl -n kuberhealthy get pod

3.2 配置额外的检查

接下来,你可以运行下面的命令,你应该会看到三个Kuberhealthy检查默认安装:

kubectl get khchecks.comcast.github.io -n kuberhealthy

  • daemonset:部署并拆除一个daemonset,以确保集群中的所有节点都能正常工作。
  • deployment:创建部署,然后触发滚动更新。测试部署是否可以通过服务访问,然后删除所有内容。此过程中的任何问题都将导致此检查报告失败。
  • dn-status-internal:验证内部集群DNS是否按预期运行。
3.3 状态查看

kuberhealthy默认会提供一个json格式的状态页面和metrics端点供我们查看状态。

之前部署之后会创建一个service,type类型为loadblance,我们实际使用可能需要修改

(1)访问json格式的状态页面

首先使用下面命令查看创建的的service

kubectl -n kuberhealthy get svc

接着访问json状态页面

(2)访问metrices

四、常见问题

Q1:使用yaml文件方式安装时,执行kubectl apply -f kuberhealthy.yaml遇到以下问题:

分析:

该yaml文件中需要拉取DockerHub的镜像包,但由于本地环境无法联网导致拉取镜像包失败

解决:

通过可以联网的主机下载镜像包后,移动到本地环境,手动加载镜像包

五、资料出处

  1. ​https://github.com/kuberhealthy/kuberhealthy​
  2. ​https://cloud.tencent.com/developer/article/1638832​
  3. ​https://cloud.tencent.com/developer/article/1661209​
  4. ​Kuberhealthy 一个还不错的健康检查 operator​
  5. ​https://mp.weixin.qq.com/s/jqROe-HQnDcHw2VPaAw-1g​
  6. ​https://github.com/kuberhealthy/kuberhealthy/blob/master/docs/K8s-KPIs-with-Kuberhealthy.md​

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值