aws codepipeline 在pipeline构建过程中使用变量

参考资料

• Action structure reference

• codebuild构建环境中的环境变量

• codepipeline中的变量

• 在codePipeline中使用变量

对于codepipeline来说，管道结构中的每个操作都有自身的结构和定义，本文主要讨论不同资源的输出变量。

基本概念

变量允许用户在执行操作时通过自定义的方式配置管道操作。变量可以通过操作执行生成，也可以在每个管道执行开始时隐式提供

变量

例如对于作为source的ecr来说，输出变量包括

• RegistryId
• RepositoryName
• ImageTag
• ImageDigest
• ImageURI

对于codebuild来说，输出变量需要在buildspec.yaml中单独导出

有的管道结构本身并没有输出变量，例如codedeploy

命名空间

命名空间提供了变量的隔离性，确保唯一引用变量，指定命名空间引用变量

#{namespace.variable_key}

命名空间有两种类型

• 预留命名空间，例如codeppeline提供的隐式命名空间codepipeline

  #{codepipeline.PipelineExecutionId}
  

• 自定义命名空间，对某个操作分配命名空间之后操作生成的变量都在该命名空间下，需要确保命名孔家年在管道定义中唯一

  #{SourceVariables.VersionId}
  

pipeline actions可用的变量

预定义的变量包括以下结构，无法添加和修改变量输出

• CodePipeline execution ID output variable
• Amazon ECR action output variables
• AWS CloudFormation StackSets action output variables
• CodeCommit action output variables
• CodeStarSourceConnection action output variables
• GitHub action output variables (GitHub action version 1)
• S3 action output variables

自定义变量键的结构有以下

• CloudFormation action output variables
• CodeBuild action output variables
• Lambda action output variables

在pipeline中输出和引用变量

新建一个3阶段的pipeline

• source阶段为codecommit，source阶段的命名空间为namespace: SourceVariables，
• build阶段在管道中配置EnvironmentVariables，可以直接引用上一步管道结构中的环境变量。此外codebuild项目中设置的环境变量和从pipeline中获得的环境变量在具体的构建过程中是一样的。
• deploy阶段指定输出到s3桶

$ aws codepipeline get-pipeline --name test-variable
metadata:
  pipelineArn: arn:aws-cn:codepipeline:cn-north-1:xxxxxxxxxx:test-variable
pipeline:
  artifactStore: ...
  name: test-variable
  roleArn: arn:aws-cn:iam::xxxxxxxxxx:role/AWSCodePipelineServiceRole
  stages:
  - actions:
  	name: Source
    - actionTypeId:
        category: Source
        provider: CodeCommit
      name: Source
      namespace: SourceVariables
  - actions:
  	name: Build
    - actionTypeId:
        category: Build
        provider: CodeBuild
      configuration:
        EnvironmentVariables: '[{"name":"pipelineid","value":"#{codepipeline.PipelineExecutionId}","type":"PLAINTEXT"},{"name":"commitid","value":"#{SourceVariables.CommitId}","type":"PLAINTEXT"}]'
        ProjectName: test-variable-build
      inputArtifacts:
      - name: SourceArtifact
      name: Build
      namespace: BuildVariables
      outputArtifacts:
      - name: BuildArtifact
  - actions:
  	name: Deploy
    - actionTypeId:
        category: Deploy
        provider: S3
      configuration:
        BucketName: zhaojiew-temptest
        Extract: 'false'
        ObjectKey: test-variable
      inputArtifacts:
      - name: BuildArtifact
      name: Deploy
      namespace: DeployVariables

source阶段

codecommit阶段输出的变量如下，不需要额外配置

build阶段

如果在codebuild中配置环境变量，是拿不到codepipeline的命名空间中的变量的

例如，通过printenv查看codebuild的环境变量

commitid-build=#{SourceVariables.CommitId}
myvariable2=bar
myvariable3=***
pipelineid=5827591f-c4b1-4497-9ec5-a92601078027
...
/codebuild/output/tmp/env.sh: line 101: export: `commitid-build': not a valid identifier

codebuild阶段可以自定义变量导出。codebuild阶段通过exported-variables将变量导出到pipeline中在下一阶段使用。这里直接将自定义的环境变量导出，设定命名空间为BuildVariables

在buildspec.yaml中增加以下配置

version: 0.2
env:
  variables:
    myvariable-shell: "shell-var"
  exported-variables:
    - myvariable2

综上可知，codebuild可以在三个地方增加环境变量

• codebuild项目
• codepipeline
• buildspec.yaml中（控制台看不到）

简单例子

manule approve中引用管道变量

查看官方的示例，在source阶段之后添加手动审批，获取对应的管道变量，当然可以直接去pipelie的构建历史中不同action中直接查看变量输出

pipeline会解析变量并生成链接和commit信息

如果变量不存在汇报以下错误，所以需要注意阶段设置正确

Invalid action configuration
An action in this pipeline failed because one or more variables could not be resolved: Action name=approve. This can result when a variable is referenced that does not exist. Validate the configuration for this action.

codebuild获取ssm parameter

默认情况下，在build阶段打印的环境变量如下

CODEBUILD_SOURCE_VERSION=arn:aws-cn:s3:::codepipeline-cn-north-1-482183469511/docker-to-ecs/SourceArti/Oh7wxuY
HOSTNAME=b13665a8f415
SBT_VERSION=1.6.2
ANDROID_HOME=/usr/local/android-sdk-linux
ANDROID_SDK_EXTRAS=extras;android;m2repository extras;google;m2repository extras;google;google_play_services
POWERSHELL_VERSION=6.2.6
ANT_DOWNLOAD_SHA512=2287dc5cfc21043c14e5413f9afb1c87c9f266ec2a9ba2d3bf2285446f6e4ccb59b558bf2e5c57911a05dfa293c7d5c7ad60ac9f744ba11406f4e6f9a27b2403
RUBY_27_VERSION=2.7.6
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/3fca43bf-4391-433a-a493-3e057990cbb3
GOLANG_13_VERSION=1.13.15
DOCKER_CHANNEL=stable
CODEBUILD_RESOLVED_SOURCE_VERSION=fe103331df4b6aca6b31649da02392049d50794e
N_SRC_DIR=/n
ANDROID_SDK_BUILD_TOOLS_28=build-tools;28.0.3
CODEBUILD_BUILD_SUCCEEDING=1
OLDPWD=/codebuild/readonly
JAVA_11_HOME=/usr/lib/jvm/java-11-amazon-corretto.x86_64
ANDROID_SDK_PLATFORM_TOOLS=platforms;android-29
CODEBUILD_PROJECT_UUID=ec86e5af-e89d-4ae1-8612-012b4e0807e8
GRADLE_VERSION=5.6.4
JRE_HOME=/usr/lib/jvm/java-11-amazon-corretto.x86_64
GITVERSION_VERSION=5.3.5
CODEBUILD_AUTH_TOKEN=06e8863a-51eb-46a3-ada0-10f95d51a01e
CODEBUILD_LOG_PATH=cc9e8db1-b0ed-4266-ae1e-3bd636cbef45
MAVEN_VERSION=3.6.3
INSTALLED_GRADLE_VERSIONS=4.10.3 5.6.4
POWERSHELL_DOWNLOAD_URL=https://github.com/PowerShell/PowerShell/releases/download/v6.2.6/powershell-6.2.6-linux-x64.tar.gz
PHP_73_VERSION=7.3.33
CODEBUILD_BUILD_URL=https://cn-north-1.console.amazonaws.cn/codebuild/home?region=cn-north-1#/builds/test-build-docker-to-ecs:cc9e8db1-b0ed-4266-ae1e-3bd636cbef45/view/new
SBT_DOWNLOAD_SHA256=637637b6c4e6fa04ab62cd364061e32b12480b09001cd23303df62b36fadd440
LOG4J_UNSAFE_VERSIONS=2.11.1 1.2.8
AWS_EXECUTION_ENV=AWS_ECS_EC2
DOCKER_BUCKET=download.docker.com
DIND_COMMIT=3b5fac462d21ca164b3778647420016315289034
MAVEN_OPTS=-Dmaven.wagon.httpconnectionManager.maxPerRoute=2
RUBY_26_VERSION=2.6.10
CODEBUILD_GOPATH=/codebuild/output/src296449416
NUGET_XMLDOC_MODE=skip
PYTHON_39_VERSION=3.9.12
PATH=/usr/local/bin/sbt/bin:/root/.goenv/shims:/root/.goenv/bin:/go/bin:/root/.phpenv/shims:/root/.phpenv/bin:/root/.pyenv/shims:/root/.pyenv/bin:/root/.rbenv/shims:/usr/local/rbenv/bin:/usr/local/rbenv/shims:/root/.dotnet/:/root/.dotnet/tools/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/tools:/usr/local/android-sdk-linux/tools:/usr/local/android-sdk-linux/tools/bin:/usr/local/android-sdk-linux/platform-tools:/codebuild/user/bin
MAVEN_HOME=/opt/maven
AWS_DEFAULT_REGION=cn-north-1
POWERSHELL_DOWNLOAD_SHA=ee5512d869ab9bd59bf17f417ff93013e0a169db91cf848ba2570d4818e05e17
PYYAML_VERSION=5.4.1
DOTNET_ROOT=/root/.dotnet
PWD=/codebuild/output/src296449416/src
CODEBUILD_BUILD_IMAGE=aws/codebuild/amazonlinux2-x86_64-standard:3.0
JAVA_HOME=/usr/lib/jvm/java-11-amazon-corretto.x86_64
GRADLE_DOWNLOADS_SHA256=abc10bcedb58806e8654210f96031db541bcd2d6fc3161e81cb0572d6a15e821 5.6.4\n336b6898b491f6334502d8074a6b8c2d73ed83b92123106bd4bf837f04111043 4.10.3
GOLANG_14_VERSION=1.14.15
CODEBUILD_FE_REPORT_ENDPOINT=https://codebuild.cn-north-1.amazonaws.com.cn/
CODEBUILD_KMS_KEY_ID=arn:aws-cn:kms:cn-north-1:xxxxxxxxxx:alias/aws/s3
GOLANG_12_VERSION=1.12.17
AWS_REGION=cn-north-1
JRE_8_HOME=/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/jre
ANDROID_SDK_MANAGER_VER=4333796
PHP_74_VERSION=7.4.29
CODEBUILD_BUILD_ARN=arn:aws-cn:codebuild:cn-north-1:xxxxxxxxxx:build/test-build-docker-to-ecs:cc9e8db1-b0ed-4266-ae1e-3bd636cbef45
PYTHON_PIP_VERSION=21.1.2
CODEBUILD_AGENT_ENDPOINT=http://127.0.0.1:7831
JDK_8_HOME=/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64
CODEBUILD_CI=true
CODEBUILD_BUILD_ID=test-build-docker-to-ecs:cc9e8db1-b0ed-4266-ae1e-3bd636cbef45
CODEBUILD_CONTAINER_NAME=default
JDK_HOME=/usr/lib/jvm/java-11-amazon-corretto.x86_64
HOME=/root
SHLVL=4
ANDROID_SDK_MANAGER_SHA256=92ffee5a1d98d856634e8b71132e8a95d96c83a63fde1099be3d86df3106def9
CODEBUILD_INITIATOR=codepipeline/docker-to-ecs
NODE_12_VERSION=12.22.12
ANDROID_SDK_BUILD_TOOLS=build-tools;29.0.3
MAVEN_DOWNLOAD_SHA512=c35a1803a6e70a126e80b2b3ae33eed961f83ed74d18fcd16909b2d44d7dada3203f1ffe726c17ef8dcca2dcaa9fca676987befeadc9b9f759967a8cb77181c0
CODEBUILD_SRC_DIR=/codebuild/output/src296449416/src
RUBY_BUILD_SRC_DIR=/usr/local/rbenv/plugins/ruby-build
DOCKER_SHA256=9ccfc39305ae1d8882d18c9c431544fca82913d6df717409ac2244ac58c4f070
CODEBUILD_BUILD_NUMBER=8
GOPATH=/go:/codebuild/output/src296449416
JDK_11_HOME=/usr/lib/jvm/java-11-amazon-corretto.x86_64
GRADLE_PATH=/gradle
DOCKER_VERSION=20.10.15
PYTHON_38_VERSION=3.8.13
DOTNET_31_SDK_VERSION=3.1.419
ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/dd120308-6d7b-43cd-a59d-258a5acb4aae
RBENV_SRC_DIR=/usr/local/rbenv
GOENV_DISABLE_GOPATH=1
ANT_VERSION=1.10.12
PYTHON_37_VERSION=3.7.13
CODEBUILD_LAST_EXIT=0
ECS_CONTAINER_METADATA_URI_V4=http://169.254.170.2/v4/dd120308-6d7b-43cd-a59d-258a5acb4aae
CODEBUILD_START_TIME=1672473450442
CODEBUILD_EXECUTION_ROLE_BUILD=
JRE_11_HOME=/usr/lib/jvm/java-11-amazon-corretto.x86_64
DOCKER_COMPOSE_VERSION=1.26.0
CODEBUILD_BMR_URL=https://CODEBUILD_AGENT:3000
JAVA_8_HOME=/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64
NODE_10_VERSION=10.24.1
ANDROID_SDK_PLATFORM_TOOLS_28=platforms;android-28
_=/usr/bin/printenv

在codebuild中配置获取ssm参数，当然也可以在管道变量中设置

配置build阶段环境变量来自ssm，需要配置额外权限

Phase context status code: Decrypted Variables Error Message: AccessDeniedException: User: arn:aws-cn:sts::xxxxxxxxxx:assumed-role/codebuild-test-variable-build-service-role/AWSCodeBuild-332a3dac-030c-4bac-9954-1faa40185aa1 is not authorized to perform: ssm:GetParameters on resource: arn:aws-cn:ssm:cn-north-1:xxxxxxxxxx:parameter/hello because no identity-based policy allows the ssm:GetParameters action

codebuild获取source阶段变量

这个之前实际上已经做过了，只需要在build阶段配置环境变量如下

EnvironmentVariables: '[{"name":"pipelineid","value":"#{codepipeline.PipelineExecutionId}","type":"PLAINTEXT"},{"name":"commitid","value":"#{SourceVariables.CommitId}","type":"PLAINTEXT"}]'

最终能够在build详情里看到解析的变量

获取并使用来自cloudforamtion的变量

Tutorial: Create a pipeline that uses variables from AWS CloudFormation deployment actions

cfn的变量输出是根据堆栈中的outputs部分指定的生辰的。

Note that the only CloudFormation action modes that generate outputs are those that result in creating or updating a stack, such as stack creation, stack updates, and change set execution.

引用语法如下

#{DeployVariables.StackName}

遗憾的是，中国区目前不支持通过pipeline部署cloudfotmation