问题
今天当我尝试部署1个新的service 到cloudrun的时候, 出现了下面错误:
gateman@DESKTOP-UIU9RFJ:~$ gcloud run deploy bq-api-service --region=europe-west2 --image=europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.0.0 --platform=managed --ingress=all --port=8080 --min-instances=0 --max-instances=2 --service-account=vm-common@jason-hsbc.iam.gserviceaccount.com --no-allow-unauthenticated
Deploying container to Cloud Run service [bq-api-service] in project [jason-hsbc] region [europe-west2]
X Deploying new service...
. Creating Revision...
. Routing traffic...
Deployment failed
ERROR: (gcloud.run.deploy) FAILED_PRECONDITION: Constraint constraints/gcp.restrictNonCmekServices violated for CreateService attempting to create revision without a CMEK key. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information.
- '@type': type.googleapis.com/google.rpc.PreconditionFailure
violations:
- description: Constraint constraints/gcp.restrictNonCmekServices violated for CreateService
attempting to create revision without a CMEK key. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints
for more information.
subject: orgpolicy:projects/jason-hsbc
type: constraints/gcp.restrictNonCmekServices
原因
原因也写得很明白, GCP的organization 新加个了policy, 所有的子项目都必须用CMEK key 加密
而我上面的部署command 并没有CMEK参数的
什么是CMEK
GCP 的 CMEK 指的是 Customer-Managed Encryption Keys(客户管理的加密密钥)。在 Google Cloud Platform(GCP)中,CMEK 允许客户自行管理用于加密其云服务数据的加密密钥。
简单来讲, 我们可以利用1个 Key 去加密保护我们的GCP component , 例如cloud run service 和 bigquery dataset, bucket 等等
创建Key Ring
参考:
https://cloud.google.com/kms/docs/monitor-ekm-usage
首先要enable CMEK api
[gateman@manjaro-x13 ~]$ gcloud services enable cloudkms.googleapis.com
[gateman@manjaro-x13 ~]$ gcloud services list --enabled | grep kms
cloudkms.googleapis.com Cloud Key Management Service (KMS) API
然后创建1个key ring
在 Google Cloud KMS(Key Management Service)中,Key Ring(密钥环)是一个逻辑容器,用于组织和管理密钥。Key Ring 用于将密钥逻辑上分组,以便更好地管理和控制密钥的访问和使用。
gcloud kms keyrings create mykeyring --location=europe-west2
查看Key Ring
[gateman@manjaro-x13 ~]$ gcloud kms keyrings list --location=europe-west2
NAME
projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring
在这个Key Ring 内创建1个CMEK (KMS Key)
[gateman@manjaro-x13 ~]$ gcloud kms keys create mycmek \
--keyring mykeyring \
--location europe-west2 \
--purpose "encryption" \
--protection-level "software"
查看CMEK
注意keyring 和 location的参数都要提供
[gateman@manjaro-x13 ~]$ gcloud kms keys list --keyring=mykeyring --location=europe-west2
NAME PURPOSE ALGORITHM PROTECTION_LEVEL LABELS PRIMARY_ID PRIMARY_STATE
projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek ENCRYPT_DECRYPT GOOGLE_SYMMETRIC_ENCRYPTION SOFTWARE 1 ENABLED
权限问题
好了, 这时1个新的CMEK 被创建 我们可以加上这个参数
–key=projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek
来部署cloud run
但是遇到了下面的权限问题
gcloud run deploy bq-api-service --region=europe-west2 --image=europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.0.0 --platform=managed --ingress=all --port=8080 --min-instances=0 --max-instances=2 --service-account=vm-common@jason-hsbc.iam.gserviceaccount.com --no-allow-unauthenticated --key=projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek
Deploying container to Cloud Run service [bq-api-service] in project [jason-hsbc] region [europe-west2]
X Deploying... Revision 'bq-api-service-00003-tav' is not ready and cannot serve traffic. Please ensure that the provided encryption key URL is correct and the project's Google Cloud Run Service Agent has the permission [cloudkms.cryptoKeyVersions.useToDecrypt] and [cloudkms.cryptoKeyVersions.useToEncrypt] on resource projects/jason-hsbc/l
ocations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek. Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek' (or it may not exist).
✓ Creating Revision...
X Routing traffic... Revision 'bq-api-service-00003-tav' is not ready and cannot serve traffic. Please ensure that the provided encryption key URL is correct and the project's Google Cloud Run Service Agent has the permission [cloudkms.cryptoKeyVersions.useToDecrypt] and [cloudkms.cryptoKeyVersions.useToEncrypt] on resource projects/jaso
n-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek. Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek' (or it may not exist).
✓ Setting IAM Policy...
Deployment failed
ERROR: (gcloud.run.deploy) Revision 'bq-api-service-00003-tav' is not ready and cannot serve traffic. Please ensure that the provided encryption key URL is correct and the project's Google Cloud Run Service Agent has the permission [cloudkms.cryptoKeyVersions.useToDecrypt] and [cloudkms.cryptoKeyVersions.useToEncrypt] on resource projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek. Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek' (or it may not exist).
原因
原因也写得很明白, Google Cloud Run Service Agent 没有 对上面的mycmeky 的两个权限。。[cloudkms.cryptoKeyVersions.useToDecrypt] and [cloudkms.cryptoKeyVersions.useToEncrypt]
检查以下这个key 的具体权限信息
果然什么都没有
[gateman@manjaro-x13 ~]$ gcloud kms keys get-iam-policy mycmek --keyring=mykeyring --location=europe-west2
etag: ACAB
什么是 Google Cloud Run Service Agent?
WTF, 我先检查一下service accounts 列表
[gateman@manjaro-x13 ~]$ gcloud iam service-accounts list
DISPLAY NAME EMAIL DISABLED
owner-test owner-test@jason-hsbc.iam.gserviceaccount.com False
App Engine default service account jason-hsbc@appspot.gserviceaccount.com False
terraform terraform@jason-hsbc.iam.gserviceaccount.com False
vm-common vm-common@jason-hsbc.iam.gserviceaccount.com False
pubsub publisher a pubsub-publisher-a@jason-hsbc.iam.gserviceaccount.com False
non-access non-access@jason-hsbc.iam.gserviceaccount.com False
terraform2 terraform2@jason-hsbc.iam.gserviceaccount.com False
Compute Engine default service account 912156613264-compute@developer.gserviceaccount.com False
没有这玩意啊, 倒是有个APP Engine default service account 和 Compute Engine default service account
然后去google
https://cloud.google.com/iam/docs/service-agents
找到了下面的信息
Google Cloud Run Service Agent
Primary service agent for run.googleapis.com.
service-PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.com
用下面命令得到project number
gcloud projects describe my-project --format=‘get(projectNumber)’
得到Google Cloud Run Service Agent
就是service-912156613264@serverless-robot-prod.iam.gserviceaccount.com
居然还describe 不出任何信息, 有点怀疑人生
[gateman@manjaro-x13 ~]$ gcloud iam service-accounts describe service-912156613264@serverless-robot-prod.iam.gserviceaccount.com
ERROR: (gcloud.iam.service-accounts.describe) PERMISSION_DENIED: Permission 'iam.serviceAccounts.get' denied on resource (or it may not exist).
- '@type': type.googleapis.com/google.rpc.ErrorInfo
domain: iam.googleapis.com
metadata:
permission: iam.serviceAccounts.get
reason: IAM_PERMISSION_DENIED
[gateman@manjaro-x13 ~]$
我姑且当它是1个隐藏的service acount
分配加解密权限给Google Cloud run App agent
[gateman@manjaro-x13 ~]$ gcloud kms keys add-iam-policy-binding mycmek \
--location=europe-west2 \
--keyring=mykeyring \
--member=serviceAccount:service-912156613264@serverless-robot-prod.iam.gserviceaccount.com \
--role=roles/cloudkms.cryptoKeyEncrypterDecrypter
Updated IAM policy for key [mycmek].
bindings:
- members:
- serviceAccount:service-912156613264@serverless-robot-prod.iam.gserviceaccount.com
role: roles/cloudkms.cryptoKeyEncrypterDecrypter
etag: BwYTpCn-w7g=
version: 1
[gateman@manjaro-x13 ~]$ gcloud kms keys get-iam-policy mycmek --keyring=mykeyring --location=europe-west2
bindings:
- members:
- serviceAccount:service-912156613264@serverless-robot-prod.iam.gserviceaccount.com
role: roles/cloudkms.cryptoKeyEncrypterDecrypter
etag: BwYTpCn-w7g=
version: 1
重新尝试部署
gateman@DESKTOP-UIU9RFJ:~$ gcloud run deploy bq-api-service --region=europe-west2 --image=europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.0.0 --platform=managed --ingress=all --port=8080 --min-instances=0 --max-instances=2 --service-account=vm-common@jason-hsbc.iam.gserviceaccount.com --no-allow-unauthenticated --key=projects/jason-hsbc/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek
Deploying container to Cloud Run service [bq-api-service] in project [jason-hsbc] region [europe-west2]
✓ Deploying... Done.
✓ Creating Revision...
✓ Routing traffic...
✓ Setting IAM Policy...
Done.
Service [bq-api-service] revision [bq-api-service-00004-dil] has been deployed and is serving 100 percent of traffic.
Service URL: https://bq-api-service-7hq3m4pdya-nw.a.run.app
这次部署成功了!