K8S使用helm部署vault集群

 一 、 Helm 安装 MySQL

备注：安装mysql只是测试vault加密的数据能否测试登录mysql，并不是把mysql作为vault数据库存储使用

1. 创建名称空间

$ kubectl create namespace vault

2. 添加mysql仓库

$ helm repo add bitnami https://charts.bitnami.com/bitnami 
"bitnami" has been added to your repositories

3. helm安装mysql

$ helm install mysql bitnami/mysql -n vault  【启动较慢等2分钟】

4. 查看mysql的pod状态

$ kubectl get pods -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
mysql-0                                 1/1     Running   0          2m58s

5. 查看mysql的service名称

$ kubectl get services -n vault
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
mysql                      ClusterIP   10.100.68.110    <none>        3306/TCP            15m
mysql-headless             ClusterIP   None             <none>        3306/TCP            15m

6. 创建一个名为ROOT_PASSWORD的变量存储 mysql root 用户密码的变量给后续使用 

$ ROOT_PASSWORD=$(kubectl get secret --namespace vault  mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode) 

二、 Helm 安装 Vault

1. 查看helm版本

$ helm version

2. 添加vault的chart仓库

$ helm repo add hashicorp https://helm.releases.hashicorp.com 
"hashicorp" has been added to your repositories

3. 测试安装 【--dry-run参数测试安装运行。查看是否可正常安装】

$ helm install vault hashicorp/vault --namespace vault --dry-run

 4. 列出vault可用版本

$ helm search repo hashicorp/vault --versions                    

 5. 在具有集成存储的 HA 模式下使用安装安装vault指定版本

$ helm install vault hashicorp/vault --namespace vault --version 0.27.0   --set='server.ha.enabled=true'   --set='server.ha.raft.enabled=true' 

 6. 查看服务状态

$ kubectl get pods -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          30s
vault-1                                 0/1     Running   0          30s
vault-2                                 0/1     Running   0          30s
mysql-0                                 1/1     Running   0          8m35s

  7. 检索Vault启动pod的状态

$ kubectl exec -n vault vault-0 -- vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            0.27.0
Storage Type       raft
HA Enabled         true
command terminated with exit code 2

三、 初始化 Vault Pod

备注：为什么要进行vault初始化，如果不初始化，vault服务pod的READY会一直处于0/1状态，只有把vault的pod进行初始化生成初始token服务才会正常启动运行

1. 初始化 Vault ，密钥以JSON格式被重定向到名为 cluster-keys.json提供给后续pod使用

$ kubectl exec -n vault vault-0 -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json > cluster-keys.json

2. 查看 cluster-keys.json文件里的初始密钥

$ cat cluster-keys.json | jq -r ".unseal_keys_b64[]"
rrUtT32GztRy/pVWmcH0ZQLCCXon/TxCgi40FL1Zzus=

3. 创建一个名为VAULT_UNSEAL_KEY捕获 Vault 初始密钥变量

$ VAULT_UNSEAL_KEY=$(cat cluster-keys.json | jq -r ".unseal_keys_b64[]")

4. 解封在vault-0 上运行的 Vault ， 就是把上面的初始化密钥倒进vault-0

$ kubectl exec -n vault vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 0.27.0
Storage Type            raft
Cluster Name            vault-cluster-16efc511
Cluster ID              649c814a-a505-421d-e4bb-d9175c7e6b38
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    31
Raft Applied Index      31

5. 检索 pod 上 Vault 的状态vault-0

$ kubectl exec -n vault vault-0  -- vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 0.27.0
Storage Type            raft
Cluster Name            vault-cluster-16efc511
Cluster ID              649c814a-a505-421d-e4bb-d9175c7e6b38
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2022-05-19T17:41:07.226862254Z
Raft Committed Index    36
Raft Applied Index      36

6.  查看初始化生成的root密钥

$ cat cluster-keys.json | jq -r ".root_token"
hvs.3VYhJODbhlQPeW5zspVvBCzD

7.  创建一个名为CLUSTER_ROOT_TOKEN 捕获 Vault的 root密钥/根令牌

$ CLUSTER_ROOT_TOKEN=$(cat cluster-keys.json | jq -r ".root_token")

8. 使用 pod 上的根令牌登录vault-0

$ kubectl exec -n vault vault-0 -- vault login $CLUSTER_ROOT_TOKEN

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.3VYhJODbhlQPeW5zspVvBCzD
token_accessor       5sy3tZm3qCQ1ai7wTDOS97XG
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

9. 列出 pod 的 Vault 集群内的所有节点vault-0

$ kubectl exec -n vault vault-0 internal-app -- vault operator raft list-peers
Node                                    Address                        State     Voter
----                                    -------                        -----     -----
09d9b35d-0336-7de7-cc94-90a1f3a0aff8    vault-0.vault-internal:8201    leader    true

10.  将 Vault 服务器加入vault-1 Vault 集群

$ kubectl exec -n vault vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
Key       Value
---       -----
Joined    true

11.  vault-1使用解封密钥解封 Vault 服务器

$ kubectl exec -n vault vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 0.27.0
Storage Type            raft
Cluster Name            vault-cluster-16efc511
Cluster ID              649c814a-a505-421d-e4bb-d9175c7e6b38
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 standby
Active Node Address     http://192.168.58.131:8200
Raft Committed Index    76
Raft Applied Index      76

12.  将 Vault 服务器加入vault-2Vault 集群

$ kubectl exec -n vault vault-2 -- vault operator raft join http://vault-0.vault-internal:8200 
Key       Value
---       -----
Joined    true

13. vault-2使用解封密钥解封 Vault 服务器 

$ kubectl exec -n vault vault-2 -- vault operator unseal $VAULT_UNSEAL_KEY
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 0.27.0
Storage Type            raft
Cluster Name            vault-cluster-16efc511
Cluster ID              649c814a-a505-421d-e4bb-d9175c7e6b38
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 standby
Active Node Address     http://192.168.58.131:8200
Raft Committed Index    76
Raft Applied Index      76

14. 列出 Vault 集群内的所有节点

$ kubectl exec -n vault vault-0 -- vault operator raft list-peers
Node                                    Address                        State       Voter
----                                    -------                        -----       -----
09d9b35d-0336-7de7-cc94-90a1f3a0aff8    vault-0.vault-internal:8201    leader      true
7078a8b7-7948-c224-a97f-af64771ad999    vault-1.vault-internal:8201    follower    true
aaf46893-0a93-17ce-115e-f57033d7f41d    vault-2.vault-internal:8201    follower    true

15. 获取vault命名空间内的所有 Pod

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          5m49s
vault-1                                 1/1     Running   0          5m48s
vault-2                                 1/1     Running   0          5m47s
vault-agent-injector-5945fb98b5-vzbqv   1/1     Running   0          5m50s
mysql-0                                 1/1     Running   0          13m54s

四、 创建 Vault 数据库角色

1. vault 启用数据库机密database

$ kubectl exec -n vault  vault-0 -- vault secrets enable database
Success! Enabled the database secrets engine at: database/

2. 使用 MySQL 数据库凭据配置数据库机密引擎【$ROOT_PASSWORD是使用上面生成的变量】

$ kubectl exec -n vault   vault-0 -- vault write database/config/mysql \
    plugin_name=mysql-database-plugin \
    connection_url="{{username}}:{{password}}@tcp(mysql.vault.svc.cluster.local:3306)/" \
    allowed_roles="readonly" \
    username="root" \
    password="$ROOT_PASSWORD"

Success! Data written to: database/config/mysql

3.  创建数据库授权角色readonly 【max_ttl令牌最长过期时间】

$ kubectl exec -n vault   vault-0 -- vault write database/roles/readonly \
    db_name=mysql \
    creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT SELECT ON *.* TO '{{name}}'@'%';" \
    default_ttl="1h" \
    max_ttl="24h"           

Success! Data written to: database/roles/readonly

 4. 从数据库角色读取凭据readonly

$ kubectl exec  -n vault  vault-0 -- vault read database/creds/readonly
Key                Value
---                -----
lease_id           database/creds/readonly/qtWlgBT1YTQEPKiXe7CrotsT
lease_duration     1h
lease_renewable    true
password           WLESe5T-RLkTj-h-lDbT
username           v-root-readonly-pk168KvLS8sc80Of

五、 配置 Kubernetes 身份验证

1. 进入到vault服务pod内部

$ kubectl exec  -n vault  --stdin=true --tty=true vault-0 -- /bin/sh
/ $ 

2.  vault 启用 Kubernetes 身份验证

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

3.  配置 Kubernetes 身份认证 【变量会自动获取相应的k8s认证值】

$ vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Success! Data written to: auth/kubernetes/config

4. 含义是新建一个策略名称devwebapp,授权database/creds/readonly路径文件的只读权限

$ vault policy write devwebapp - <<EOF
path "database/creds/readonly" {
  capabilities = ["read"]
}
EOF

5.  创建 Kubernetes 身份验证角色名称为 devweb-app【bound_service_account_names 表示RBAC用户名称，bound_service_account_namespaces表示RBAC用户所在的名称空间，policies表示使用的策略名称，ttl表示 Kubernetes 身份验证角色超时时间】

$ vault write auth/kubernetes/role/devweb-app \
      bound_service_account_names=internal-app \
      bound_service_account_namespaces=vault \
      policies=devwebapp \
      ttl=24h

Success! Data written to: auth/kubernetes/role/devweb-app

六、 启动应用程序验证

1. 创建要使用的RBAC用户

$ cat > internal-app.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-app
  namespace: vault
EOF

2. 创建应用程序来测试能否从vault中读取用户密码

$ kubectl apply -f internal-app.yaml
cat > devwebapp.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: devwebapp
  namespace: vault
  labels:
    app: devwebapp
spec:
  replicas: 1  
  selector:
    matchLabels:
      app: devwebapp
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true" 
        vault.hashicorp.com/agent-cache-enable: "true"
        vault.hashicorp.com/role: "devweb-app"             #【Kubernetes 身份验证角色名称】
        vault.hashicorp.com/agent-inject-secret-database-connect.sh: "database/creds/readonly"  #【设置密码保存的脚本名称database-connect.sh，以及要访问的密钥文件路径】
        vault.hashicorp.com/agent-inject-template-database-connect.sh: |
          {{- with secret "database/creds/readonly" -}}
          mysql -h my-release-mysql.vault.svc.cluster.local --user={{ .Data.username }} --password={{ .Data.password }} my_database

          {{- end -}} 【给database-connect.sh脚本传入特定的值】

      labels:
        app: devwebapp
    spec:
      serviceAccountName: internal-app
      containers:
        - name: devwebapp
          image: nginx:latest  
          ports:
            - containerPort: 80  
EOF

3. 执行yaml文件 【服务启动时会直接以Sidecar形式跑两个容器到服务中，是为了共享密钥文件到主应用的共享目录中】

$ kubectl apply -f devwebapp.yaml
deployment/devwebapp created

4.  获取默认命名空间内的所有 Pod

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
devwebapp                               2/2     Running   0          36s
mysql-0                                 1/1     Running   0          25m32s
vault-0                                 1/1     Running   0          17m40s
vault-1                                 1/1     Running   0          17m40s
vault-2                                 1/1     Running   0          17m40s
vault-agent-injector-76fff8f7c6-lk6gz   1/1     Running   0          17m40s

5.  获取容器路径 /vault/secrets/database-connect.sh 的机密值

kubectl exec -n vault --stdin=true \
    --tty=true devwebapp \
    --container devwebapp \
    -- cat /vault/secrets/database-connect.sh

出现如下所示表示获取密钥成功
mysql -h my-release-mysql.default.svc.cluster.local --user=v-kubernetes-readonly-zpqRzAee2b --password=Jb4epAXSirS2s-pnrI9- my_database

以上完成了vault部署到测试的全部流程

官方文档参考：Vault installation to Amazon Elastic Kubernetes Service via Helm | Vault | HashiCorp Developer

vault 中文参考文档：

token renew · 《Vault 中文手册》