【k8s admission 学习】连接 k8s 集群 InClusterConfig 和 ServiceAccount

参考

• k8s-pod-pod内程序操作k8s资源(InClusterConfig)

访问集群过程

当我们开发一个程序，需要访问k8s集群中的pod、deployment等资源时，会使用k8s.io/client-go模块，在使用这个模块时，我们要有如下几步：

1. 获取config对象
   • clientcmd.BuildConfigFromFlags 根据config路径获取config
   • rest.InClusterConfig 直接使用pod中自带的token等内容
2. 获取k8s client
3. 使用k8s client获取k8s资源

package main

import (
    "context"
    "fmt"
    v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/util/homedir"
    "os"
    "path/filepath"
)

func main() {
    kubeConfig, err := CreateKubeConfig()
    if err != nil {
        panic(err)

    }
    kubeClient, err := kubernetes.NewForConfig(kubeConfig)
    if err != nil {
        panic(err)
    }
    //获取pod资源
    kubeClient.CoreV1().Pods("").List(context.Background(),v1.ListOptions{})


}
func PathExists(path string) (bool, error) {
    _, err := os.Stat(path)
    if err == nil {
        return true, nil
    }
    if os.IsNotExist(err) {
        return false, nil
    }
    return false, err
}

func CreateKubeConfig() (*rest.Config, error) {
    kubeConfigPath := ""
    if home := homedir.HomeDir(); home != "" {
        kubeConfigPath = filepath.Join(home, ".kube", "config")
    }
    fileExist, err := PathExists(kubeConfigPath)
    if err != nil {
        return nil, fmt.Errorf("justify kubeConfigPath exist err,err:%v", err)
    }
    //.kube/config文件存在，就使用文件
    //这里主要是本地测试
    if fileExist {
        config, err := clientcmd.BuildConfigFromFlags("", kubeConfigPath)
        if err != nil {
            return nil, err
        }
        return config, nil
    } else {
        //当程序以pod方式运行时，就直接走这里的逻辑
        config, err := rest.InClusterConfig()
        if err != nil {
            return nil, err
        }
        return config, nil
    }
}

InClusterConfig

1.创建serviceAccount

• 要想操作k8s的相关资源，需要给某个serviceAccount授权

• 如上：我们要操作pod资源，就要创建如下资源

• 创建如下资源后，opPodServiceAccount这个serviceAccout就有操作pod的权限了

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: opPodClusterRole
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - list
      - watch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: opPodClusterRoleBinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: opPodClusterRole
subjects:
  - kind: ServiceAccount
    name: opPodServiceAccount
    namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: opPodServiceAccount
  namespace: default

2.指定运行程序的pod使用上面的ServiceAccount

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    operator: live-media-watch-pod
  name: live-media-watch-pod
  namespace: bixin-system
spec:
  replicas: 1
  selector:
    matchLabels:
      operator: live-media-watch-pod
  strategy:
    rollingUpdate:
      maxSurge: 100%
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        operator: live-media-watch-pod
    spec:
      containers:
        - image: ******.**.com/k8s/live-media-watch-pod:202201211654
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 10
            httpGet:
              path: healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
          name: live-media-watch-pod
          readinessProbe:
            failureThreshold: 10
            httpGet:
              path: healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3
      serviceAccount: live-media-watch-pod # 使用上面创建的 ServiceAccount

• 这样之后，我们查看pod的yaml，会看到pod自动就多了一个volums，来自live-media-watch-pod-token-fhvk2 secret。

• 而且该secret挂载在了/var/run/secrets/kubernetes.io/serviceaccount路径下。

ServiceAccount 创建时会自动创建个 secret 存储 token，用于访问 apiserver

ServiceAccount 挂载到 pod 后，一般会在/var/run/secrets/kubernetes.io/serviceaccount路径下（kubectl describe pod podname 会看到具体的挂载路径）

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: ack.privileged
  creationTimestamp: "2022-01-21T08:57:44Z"
  generateName: live-media-watch-pod-6f688c8b98-
  labels:
    operator: live-media-watch-pod
    pod-template-hash: 6f688c8b98
  name: live-media-watch-pod-6f688c8b98-w9txp
  namespace: bixin-system
  ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: live-media-watch-pod-6f688c8b98
      uid: 79843c47-296c-4bbb-8ff3-82d7fe74719b
  resourceVersion: "83659547"
  uid: 23ef3ff1-bef6-4d7d-9e75-faea932f7919
spec:
  containers:
    - image: **.**.com/k8s/live-media-watch-pod:202201211654
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 10
        httpGet:
          path: healthz
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 30
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 3
      name: live-media-watch-pod
      readinessProbe:
        failureThreshold: 10
        httpGet:
          path: healthz
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 30
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 3
      volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount  # 此处即为 挂载在 pod 内的路径
          name: live-media-watch-pod-token-fhvk2 
          readOnly: true
  serviceAccount: live-media-watch-pod
  serviceAccountName: live-media-watch-pod
  volumes:
    - name: live-media-watch-pod-token-fhvk2
      secret:
        defaultMode: 420
        secretName: live-media-watch-pod-token-fhvk2

进到pod中查看

3.查看InClusterConfig源码

也是从/var/run/secrets/kubernetes.io/serviceaccount这个路径中获取token

实际上k8s中也有一个默认的serviceAccount：default，同样挂载在pod中的上述路径下，只是这个默认的serviceAccount权限很小，所以才会需要创建自定义的serviceAccount

apiVersion: v1
kind: Pod
metadata:
  name: live-media-agent-cpu-6kzfc
  namespace: ops
spec:
  containers:
      image: **.*.com/ops/live-media-agent:202201221010
      imagePullPolicy: IfNotPresent
      name: live-media-agent-cpu
      volumeMounts:
        - mountPath: /data/config
          name: vol1
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: default-token-5jqzn
          readOnly: true
  serviceAccount: default
  serviceAccountName: default
  volumes:
    - name: default-token-5jqzn
      secret:
        defaultMode: 420
        secretName: default-token-5jqzn