aws codebuild 自定义构建环境和本地构建

参考资料

• Extending AWS CodeBuild with Custom Build Environments

• Docker in custom image sample for CodeBuild

codebuild自定义构建环境

在创建codebuild项目的时候发现

构建环境是 Docker 映像，其中包含构建和测试项目所需的所有内容的完整文件系统

用rust写一个简单的helloworld，添加一段单元测试

#[derive(Debug)]
struct Rectangle {
    width: u32,
    height: u32,
}

impl Rectangle {
    fn can_hold(&self, other: &Rectangle) -> bool {
        self.width > other.width && self.height > other.height
    }
}

pub fn add_tow(a: i32) -> i32 {
    a + 2
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn larger_can_hold_smaller() {
        let larger = Rectangle { width: 8, height: 7 };
        let smaller = Rectangle { width: 5, height: 1 };

        assert!(larger.can_hold(&smaller));
    }

    use crate::add_tow;

    #[test]
    fn if_add_twos() {
        assert_eq!(4, add_tow(2));
    }
    
    #[test]
    fn if_add_twos_info() {
        let result =  add_tow(2);
        // assert!(5 == result,"failed to add two for 3, get wrong value {}",result);
        assert_eq!(5,result,"failed to add two for 3, get wrong value {}",result);
    }
}

创建buildspec.yml配置

version: 0.2

phases:
  install:
    commands:
      - echo "install stage"
  pre_build:
    commands:
      - echo "pre_build stage"
      - cargo version
  build:
    commands:
      - echo "build stage"
      - cargo build
  post_build:
  	commands:
      - echo "post_build test stage"
      - cargo test
artifacts:
  files:
    - target/release/rustdemo
  discard-paths: yes

创建codebuild项目如下

$ aws codebuild create-project --cli-input-json file://buildfile.json
// buildfile.json
{
    "name": "test-codebuild-rust",
    "source": {
        "type": "CODECOMMIT",
        "location": "https://git-codecommit.cn-north-1.amazonaws.com.cn/v1/repos/test-codebuild-rust",
        "gitCloneDepth": 1,
        "gitSubmodulesConfig": {
            "fetchSubmodules": false
        },
        "insecureSsl": false
    },
    "sourceVersion": "refs/heads/master^{8f5b871be5bf92ef70155801b2f35e4af0779632}",
    "artifacts": {
        "type": "S3",
        "location": "zhaojiew-test",
        "path": "",
        "namespaceType": "NONE",
        "name": "rustdemo",
        "packaging": "ZIP",
        "overrideArtifactName": false,
        "encryptionDisabled": false
    },
    "environment": {
        "type": "LINUX_CONTAINER",
        "image": "xxxxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/myrust:latest",
        "computeType": "BUILD_GENERAL1_SMALL",
        "environmentVariables": [],
        "privilegedMode": false,
        "imagePullCredentialsType": "CODEBUILD"
    },
    "serviceRole": "arn:aws-cn:iam::xxxxxxxxx:role/service-role/codebuild-test-codebuild-rust-service-role",
    "timeoutInMinutes": 60,
    "queuedTimeoutInMinutes": 480,
    "encryptionKey": "arn:aws-cn:kms:cn-north-1:xxxxxxxxx:alias/aws/s3",
    "logsConfig": {
        "cloudWatchLogs": {
            "status": "ENABLED"
        },
        "s3Logs": {
            "status": "DISABLED",
            "encryptionDisabled": false
        }
    }
}

开始构建

aws codebuild start-build --project-name test-codebuild-rust

第一次在PROVISIONING阶段报错，由于指定仓库为ecr，需要给codebuild-test-codebuild-rust-service-role权限拉取ecr镜像

BUILD_CONTAINER_UNABLE_TO_PULL_IMAGE: Unable to pull customer's container image. CannotPullContainerError: Error response from daemon: pull access denied for xxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/myrust, repository does not exist or may require 'docker login': denied: User: CodeBuild

第二次出现POST_BUILD预期错误，即测试失败

COMMAND_EXECUTION_ERROR: Error while executing command: cargo test. Reason: exit status 101

第三次成功build

此外，在选择build角色的时候，可以选择两个不同的角色

这两个角色的区别在于一个是自动创建的servicerole，另一个是该项目的servicerole

测试过程中使用默认servicerole，即使权限足够也会出现以下报错

BUILD_CONTAINER_UNABLE_TO_PULL_IMAGE: Unable to pull customer's container image. CannotPullContainerError: Error response from daemon: pull access denied for 037047667284.dkr.ecr.cn-north-1.amazonaws.com.cn/myrust, repository does not exist or may require 'docker login': denied: User: CodeBuild

codebuild本地构建

https://docs.aws.amazon.com/codebuild/latest/userguide/use-codebuild-agent.html

本地构建意味着虽然codebuild项目仍旧在aws环境中托管，但是实际的构建环境转移到本地

先决条件

• 安装git和docker
• 安装codebuild代理

设置build的环境镜像，可以直接使用托管的ecr或手动按需build，构建环境中支持的命令和环境可以参考dockerbuild内容

aws codebuild doceker image

codebuild托管的构建环境镜像有以下

• standard 4.0
• standard 5.0
• standard 6.0
• amazonlinux2-x86_64-standard:3.0
• amazonlinux2-x86_64-standard:4.0
• amazonlinux2-aarch64-standard:1.0
• amazonlinux2-aarch64-standard:2.0

docker pull public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:3.0

下载codebuild代理image（x86_64 版本），之后使用构建脚本启动，该代理为构建脚本的默认代理，如果没有会自动下载

docker pull public.ecr.aws/codebuild/local-builds:latest

下载构建脚本

构建脚本的具体内容，https://github.com/aws/aws-codebuild-docker-images/blob/master/local_builds/codebuild_build.sh

$ curl -O  https://raw.githubusercontent.com/aws/aws-codebuild-docker-images/master/local_builds/codebuild_build.sh
$ chmod +x codebuild_build.sh
$ ./codebuild_build.sh
The image name flag (-i) must be included for a build to run
The artifact directory (-a) must be included for a build to run
# -i 指定build的image环境
# -a 指定构建的输出目录
# -s 需要构建项目目录，默认为当前目录
# -l 覆盖默认的代理镜像

在本地新建maven项目并创建buildspec.yml

version: 0.2

phases:
  install:
    runtime-versions:
      java: corretto8
  pre_build:
    commands:
      - echo Initializing environment
  build:
    commands:
      - echo Build started on `date`
      - mvn compile
  post_build:
    commands:
      - echo Build completed on `date`
      - mvn package
artifacts:
  files:
    - appspec.yml
    - scripts/**/*
    - target/unicorn-web-project.war
  discard-paths: no

开始构建

./codebuild_build.sh -i public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:3.0 -a artificats

构建输出如下

docker run -it -v /var/run/docker.sock:/var/run/docker.sock -e "IMAGE_NAME=public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:3.0" -e "ARTIFACTS=/home/ec2-user/efs/test-build-javaweb/artificats" -e "SOURCE=/home/ec2-user/efs/test-build-javaweb" -e "INITIATOR=ec2-user" public.ecr.aws/codebuild/local-builds:latest

Removing network agent-resources_default
Removing volume agent-resources_source_volume
Removing volume agent-resources_user_volume
Creating network "agent-resources_default" with the default driver
Creating volume "agent-resources_source_volume" with local driver
Creating volume "agent-resources_user_volume" with local driver
Pulling agent (amazon/aws-codebuild-local:latest)...
latest: Pulling from amazon/aws-codebuild-local
Pulling build (public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:3.0)...
3.0: Pulling from codebuild/amazonlinux2-x86_64-standard
Creating agent-resources_agent_1 ... done
Creating agent-resources_build_1 ... done
Attaching to agent-resources_agent_1, agent-resources_build_1
agent_1  | [Container] 2023/02/17 06:09:22 Waiting for agent ping
agent_1  | [Container] 2023/02/17 06:09:22 Waiting for DOWNLOAD_SOURCE
agent_1  | [Container] 2023/02/17 06:09:24 Phase is DOWNLOAD_SOURCE
agent_1  | [Container] 2023/02/17 06:09:24 CODEBUILD_SRC_DIR=/codebuild/output/src116956819/src
agent_1  | [Container] 2023/02/17 06:09:24 yamlDoc
agent_1  | [Container] 2023/02/17 06:09:24 YAML location is /codebuild/output/srcDownload/src/buildspec.yml
agent_1  | [Container] 2023/02/17 06:09:24 No commands found for phase name: install
agent_1  | [Container] 2023/02/17 06:09:24 Processing environment variables
agent_1  | [Container] 2023/02/17 06:09:24 Running command echo "Installing corretto(OpenJDK) version 8 ..."