google cloud run service - 用CMEK 加密容器

问题

今天当我尝试部署1个新的service 到cloudrun的时候， 出现了下面错误：

gateman@DESKTOP-UIU9RFJ:~$ gcloud run deploy bq-api-service --region=europe-west2 --image=europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.0.0 --platform=managed --ingress=all --port=8080 --min-instances=0 --max-instances=2 --service-account=vm-common@jason-hsbc.iam.gserviceaccount.com --no-allow-unauthenticated
Deploying container to Cloud Run service [bq-api-service] in project [jason-hsbc] region [europe-west2]
X Deploying new service...                                                                                                                                                                                                                                                                                                                           
  . Creating Revision...                                                                                                                                                                                                                                                                                                                             
  . Routing traffic...                                                                                                                                                                                                                                                                                                                               
Deployment failed                                                                                                                                                                                                                                                                                                                                    
ERROR: (gcloud.run.deploy) FAILED_PRECONDITION: Constraint constraints/gcp.restrictNonCmekServices violated for CreateService attempting to create revision without a CMEK key. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information.
- '@type': type.googleapis.com/google.rpc.PreconditionFailure
  violations:
  - description: Constraint constraints/gcp.restrictNonCmekServices violated for CreateService
      attempting to create revision without a CMEK key. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints
      for more information.
    subject: orgpolicy:projects/jason-hsbc
    type: constraints/gcp.restrictNonCmekServices

原因

原因也写得很明白， GCP的organization 新加个了policy, 所有的子项目都必须用CMEK key 加密
而我上面的部署command 并没有CMEK参数的

什么是CMEK

GCP 的 CMEK 指的是 Customer-Managed Encryption Keys（客户管理的加密密钥）。在 Google Cloud Platform（GCP）中，CMEK 允许客户自行管理用于加密其云服务数据的加密密钥。

简单来讲， 我们可以利用1个 Key 去加密保护我们的GCP component ， 例如cloud run service 和 bigquery dataset, bucket 等等

创建Key Ring

参考：
https://cloud.google.com/kms/docs/monitor-ekm-usage

首先要enable CMEK api

[gateman@manjaro-x13 ~]$ gcloud services enable cloudkms.googleapis.com
[gateman@manjaro-x13 ~]$ gcloud services list --enabled | grep kms
cloudkms.googleapis.com              Cloud Key Management Service (KMS) API

然后创建1个key ring

在 Google Cloud KMS（Key Management Service）中，Key Ring（密钥环）是一个逻辑容器，用于组织和管理密钥。Key Ring 用于将密钥逻辑上分组，以便更好地管理和控制密钥的访问和使用。

gcloud kms keyrings create mykeyring --location=europe-west2

查看Key Ring

[gateman@manjaro-x13 ~]$ gcloud kms keyrings list --location=europe-west2
NAME
projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring

在这个Key Ring 内创建1个CMEK (KMS Key)

[gateman@manjaro-x13 ~]$ gcloud kms keys create mycmek \
    --keyring mykeyring \
    --location europe-west2 \
    --purpose "encryption" \
    --protection-level "software"

查看CMEK

注意keyring 和 location的参数都要提供

[gateman@manjaro-x13 ~]$ gcloud kms keys list --keyring=mykeyring --location=europe-west2
NAME                                                                             PURPOSE          ALGORITHM                    PROTECTION_LEVEL  LABELS  PRIMARY_ID  PRIMARY_STATE
projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek  ENCRYPT_DECRYPT  GOOGLE_SYMMETRIC_ENCRYPTION  SOFTWARE                  1           ENABLED

权限问题

好了， 这时1个新的CMEK 被创建 我们可以加上这个参数
–key=projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek

来部署cloud run

但是遇到了下面的权限问题

gcloud run deploy bq-api-service --region=europe-west2 --image=europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.0.0 --platform=managed --ingress=all --port=8080 --min-instances=0 --max-instances=2 --service-account=vm-common@jason-hsbc.iam.gserviceaccount.com --no-allow-unauthenticated --key=projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek
Deploying container to Cloud Run service [bq-api-service] in project [jason-hsbc] region [europe-west2]
X Deploying... Revision 'bq-api-service-00003-tav' is not ready and cannot serve traffic. Please ensure that the provided encryption key URL is correct and the project's Google Cloud Run Service Agent has the permission [cloudkms.cryptoKeyVersions.useToDecrypt] and [cloudkms.cryptoKeyVersions.useToEncrypt] on resource projects/jason-hsbc/l
ocations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek. Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek' (or it may not exist).                                                                                                         
  ✓ Creating Revision...                                                                                                                                                                                                                                                                                                                             
  X Routing traffic... Revision 'bq-api-service-00003-tav' is not ready and cannot serve traffic. Please ensure that the provided encryption key URL is correct and the project's Google Cloud Run Service Agent has the permission [cloudkms.cryptoKeyVersions.useToDecrypt] and [cloudkms.cryptoKeyVersions.useToEncrypt] on resource projects/jaso
  n-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek. Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek' (or it may not exist).                                                                                               
  ✓ Setting IAM Policy...                                                                                                                                                                                                                                                                                                                            
Deployment failed                                                                                                                                                                                                                                                                                                                                    
ERROR: (gcloud.run.deploy) Revision 'bq-api-service-00003-tav' is not ready and cannot serve traffic. Please ensure that the provided encryption key URL is correct and the project's Google Cloud Run Service Agent has the permission [cloudkms.cryptoKeyVersions.useToDecrypt] and [cloudkms.cryptoKeyVersions.useToEncrypt] on resource projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek. Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek' (or it may not exist).

原因

原因也写得很明白， Google Cloud Run Service Agent 没有 对上面的mycmeky 的两个权限。。[cloudkms.cryptoKeyVersions.useToDecrypt] and [cloudkms.cryptoKeyVersions.useToEncrypt]

检查以下这个key 的具体权限信息
果然什么都没有

[gateman@manjaro-x13 ~]$ gcloud kms keys get-iam-policy mycmek --keyring=mykeyring --location=europe-west2
etag: ACAB

什么是 Google Cloud Run Service Agent？

WTF, 我先检查一下service accounts 列表

[gateman@manjaro-x13 ~]$ gcloud iam service-accounts list
DISPLAY NAME EMAIL DISABLED
owner-test owner-test@jason-hsbc.iam.gserviceaccount.com False
App Engine default service account jason-hsbc@appspot.gserviceaccount.com False
terraform terraform@jason-hsbc.iam.gserviceaccount.com False
vm-common vm-common@jason-hsbc.iam.gserviceaccount.com False
pubsub publisher a pubsub-publisher-a@jason-hsbc.iam.gserviceaccount.com False
non-access non-access@jason-hsbc.iam.gserviceaccount.com False
terraform2 terraform2@jason-hsbc.iam.gserviceaccount.com False
Compute Engine default service account 912156613264-compute@developer.gserviceaccount.com False

没有这玩意啊， 倒是有个APP Engine default service account 和 Compute Engine default service account

然后去google
https://cloud.google.com/iam/docs/service-agents

找到了下面的信息
Google Cloud Run Service Agent
Primary service agent for run.googleapis.com.
service-PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.com

用下面命令得到project number
gcloud projects describe my-project --format=‘get(projectNumber)’

得到Google Cloud Run Service Agent
就是service-912156613264@serverless-robot-prod.iam.gserviceaccount.com

居然还describe 不出任何信息， 有点怀疑人生

[gateman@manjaro-x13 ~]$ gcloud iam service-accounts describe service-912156613264@serverless-robot-prod.iam.gserviceaccount.com

ERROR: (gcloud.iam.service-accounts.describe) PERMISSION_DENIED: Permission 'iam.serviceAccounts.get' denied on resource (or it may not exist).
- '@type': type.googleapis.com/google.rpc.ErrorInfo
  domain: iam.googleapis.com
  metadata:
    permission: iam.serviceAccounts.get
  reason: IAM_PERMISSION_DENIED
[gateman@manjaro-x13 ~]$ 

我姑且当它是1个隐藏的service acount

分配加解密权限给Google Cloud run App agent

[gateman@manjaro-x13 ~]$ gcloud kms keys add-iam-policy-binding mycmek \
  --location=europe-west2 \
  --keyring=mykeyring \
  --member=serviceAccount:service-912156613264@serverless-robot-prod.iam.gserviceaccount.com \
  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter
Updated IAM policy for key [mycmek].
bindings:
- members:
  - serviceAccount:service-912156613264@serverless-robot-prod.iam.gserviceaccount.com
  role: roles/cloudkms.cryptoKeyEncrypterDecrypter
etag: BwYTpCn-w7g=
version: 1

[gateman@manjaro-x13 ~]$ gcloud kms keys get-iam-policy mycmek --keyring=mykeyring --location=europe-west2
bindings:
- members:
  - serviceAccount:service-912156613264@serverless-robot-prod.iam.gserviceaccount.com
  role: roles/cloudkms.cryptoKeyEncrypterDecrypter
etag: BwYTpCn-w7g=
version: 1

重新尝试部署

gateman@DESKTOP-UIU9RFJ:~$ gcloud run deploy bq-api-service --region=europe-west2 --image=europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.0.0 --platform=managed --ingress=all --port=8080 --min-instances=0 --max-instances=2 --service-account=vm-common@jason-hsbc.iam.gserviceaccount.com --no-allow-unauthenticated --key=projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek
Deploying container to Cloud Run service [bq-api-service] in project [jason-hsbc] region [europe-west2]
✓ Deploying... Done.                                                                                                                                                                                                                                                                                                                                 
  ✓ Creating Revision...                                                                                                                                                                                                                                                                                                                             
  ✓ Routing traffic...                                                                                                                                                                                                                                                                                                                               
  ✓ Setting IAM Policy...                                                                                                                                                                                                                                                                                                                            
Done.                                                                                                                                                                                                                                                                                                                                                
Service [bq-api-service] revision [bq-api-service-00004-dil] has been deployed and is serving 100 percent of traffic.
Service URL: https://bq-api-service-7hq3m4pdya-nw.a.run.app

这次部署成功了！