使用Sam和Jenkins自动执行Lambda图层

吴雄辉

于 2020-10-09 06:37:44 发布

阅读量371

点赞数

原文链接：https://medium.com/fundbox-engineering/automating-lambda-layers-with-sam-and-jenkins-53ea8f002c0a

版权

Our data orchestration platform is using several lambda functions to manage, audit, and monitor data processes. Our Python lambda functions share some logic such as logging, SQS utilities, slack communication code, and more.

我们的数据编排平台正在使用几个lambda函数来管理，审计和监视数据过程。我们的Python lambda函数共享一些逻辑，例如日志记录，SQS实用程序，松弛的通信代码等等。

To reuse Python modules we used to copy a bunch of .py files containing our shared modules into multiple lambdas. This, obviously is not a good practice as we had to build, pack, and deploy all lambdas upon changes in shared modules.

为了重用Python模块，我们曾经将一堆包含共享模块的.py文件复制到多个lambda中。显然，这不是一个好习惯，因为我们必须根据共享模块的更改来构建，打包和部署所有lambda。

Lambda图层 (Lambda Layers)

Instead of duplicating code, we decided to share code between lambdas using Lambda Layers.

我们决定不使用Lambda图层在Lambda之间共享代码，而不是复制代码。

You can configure your Lambda function to pull in additional code and content in the form of layers. A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package.

您可以配置Lambda函数以层的形式引入其他代码和内容。层是一个ZIP归档文件，其中包含库，自定义运行时或其他依赖项。通过层，您可以在函数中使用库，而无需在部署包中包括它们。

In addition:

此外：

Layers let you keep your deployment package small, which makes development easier. You can avoid errors that can occur when you install and package dependencies with your function code. For Node.js, Python, and Ruby functions, you can develop your function code in the Lambda console as long as you keep your deployment package under 3 MB.

分层使您可以减小部署程序包的大小，从而使开发更加容易。您可以避免在使用功能代码安装和打包依赖项时发生的错误。对于Node.js，Python和Ruby函数，只要将部署程序包的大小保持在3 MB以内，就可以在Lambda控制台中开发函数代码。

创建一个图层 (Creating a Layer)

Assuming we have a Python3.7 compatible lambda function, and we want to add requests version 2.24.0 package to our lambda inside a layer. We need to do the following:

假设我们具有与Python3.7兼容的lambda函数，并且我们想将2.24.0版的请求包添加到我们的lambda层中。我们需要执行以下操作：

Add requests 2.24.0 into a requirements.txt file
将请求2.24.0添加到requirements.txt文件中
Install the requirements using pip into a destination folder:
使用pip将需求安装到目标文件夹中：

Install the requirements using pip into a destination folder:pip install -r requirements.txt -t python/lib/python3.7/site-packages/
使用pip将需求安装到目标文件夹中： pip install -r requirements.txt -t python/lib/python3.7/site-packages/
Zip that folder as layer.zip
将该文件夹压缩为layer.zip
Create a layer in the AWS Lambda console and upload the zip
在AWS Lambda控制台中创建一个图层并上传zip

Creation of Lambda Layer in AWS Lambda console — Creating a Lambda Layer in AWS Lambda Console

障碍物(Obstacles)

Binaries — as you probably guessed, your local python binaries might be different from the Lambda runtime.
二进制文件-您可能已经猜到了，本地python二进制文件可能与Lambda运行时不同。

In other words, code running on your laptop can sometimes behave differently in lambda function due to runtime differences. Meaning that
换句话说，由于运行时差异，笔记本电脑上运行的代码有时在lambda函数中的行为会有所不同。意思是

you need a Lambda Python3.7 runtime when you install your packages.
安装软件包时，需要Lambda Python3.7运行时。

To work around this, you can use a local Lambda runtime by running a
要解决此问题，您可以通过运行以下命令来使用本地Lambda运行时

docker image. Matthew Vielkind wrote a wonderful blog post describing the steps in detail.
码头工人图像。 Matthew Vielkind写了一篇很棒的博客文章，详细描述了这些步骤。
Time — even if you’ll use Docker to build the right binaries it will be time-consuming.
时间—即使您将使用Docker构建正确的二进制文件，也将非常耗时。

转到AWS SAM CLI (Pivoting to AWS SAM CLI)

This process can be much easier if you’ll use AWS SAM CLI.

如果您将使用AWS SAM CLI，则此过程会更加容易。

Using SAM CLI you can build any lambda layer (and many other resources such as Lambda functions, API Gateway APIs, and more) without taking care of docker images and runtime issues (Docker is necessary for SAM CLI).

使用SAM CLI，您可以构建任何lambda层(以及许多其他资源，例如Lambda函数，API Gateway API等)，而无需考虑docker映像和运行时问题(对于SAM CLI，Docker是必需的)。

In addition, SAM CLI is using CloudFormation to manage your resources, this will make it easier for you to update your layer (or other AWS resources) or delete it in the future.

此外，SAM CLI使用CloudFormation来管理您的资源，这将使您将来更轻松地更新图层(或其他AWS资源)或将其删除。

我们如何做？ (How Are We Doing It?)

Image for post — Building Lambda Layers using Jenkins and SAM at Fundbox

Steps:

脚步：

A Jenkins job is used to pull a requrements.txt file from a Github repo.
Jenkins作业用于从Github存储库中提取requrements.txt文件。
Jenkins runs SAM CLI commands to build, pack and deploy the layer.
Jenkins运行SAM CLI命令来构建，打包和部署该层。
SAM CLI installs packages from PyPI, zip the packages, upload the zip to S3, and uses a CloudFormation template to deploy the layer to AWS.
SAM CLI从PyPI安装软件包，压缩软件包，将zip上传到S3，然后使用CloudFormation模板将层部署到AWS。
AWS Lambda service will download the layer code from S3 and create the layer
AWS Lambda服务将从S3下载层代码并创建层

SAM CLI is using templates (YAML files) to describe the resources in AWS. If you create the basic SAM project using SAM CLI you end up with a template similar to this:

SAM CLI使用模板(YAML文件)来描述AWS中的资源。如果使用SAM CLI创建基本的SAM项目，则最终会得到一个类似于以下内容的模板：

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    sam-app
Sample SAM Template for sam-app
Globals:
    Function:
        Timeout: 3
Resources:
HelloWorldFunction:
    Type: AWS::Serverless::Function 
    Properties:
        CodeUri: hello_world/build/
        Handler: app.lambda_handler
        Runtime: python3.7
        Events:
            HelloWorld:
                Type: Api 
                Properties:
                    Path: /hello
                    Method: get

HelloWorldFunction is the resource we manage using SAM CLI, it a Python3.7 method and it has 1 trigger event from API Gateway.

HelloWorldFunction是我们使用SAM CLI管理的资源，它是Python3.7方法，并且具有来自API网关的1个触发事件。

SAM模板 (SAM Template)

In our Lambda Layer case, the template is:

在我们的Lambda图层案例中，模板为：

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 'Layer built from requirements.txt'


Parameters:
    LayerHumanName:
        Description: 'Human readable layer name'
        Type: 'String'


    Description:
        Description: 'Description of the layer'
        Type: 'String'


Resources:
  PackagesLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: !Ref LayerHumanName
      Description: !Ref Description
      ContentUri: 'layer/'
    CompatibleRuntimes:
      - python3.7
    LicenseInfo: MIT
    RetentionPolicy: Retain
    Metadata:
      BuildMethod: python3.7


  BasicParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Type: String
      Name: !Sub '/dataops/lambda-layer/${LayerHumanName}/latest'
      Value: !Ref PackagesLayer
      Description: SSN parameter for the latest arn of the layer

In order to get from a template to a real layer, we need to run a few SAM CLI commands. As can be seen in line #6 and #10 our template requires 2 parameters LayerHumanName and Description, these parameters are used later in the code.

为了从模板到真实层，我们需要运行一些SAM CLI命令。从第6行和第10行可以看出，我们的模板需要2个参数LayerHumanName和Description，这些参数稍后在代码中使用。

As automation is our goal, we created a simple .sh script that will execute SAM CLI commands for us:

由于自动化是我们的目标，因此我们创建了一个简单的.sh脚本，它将为我们执行SAM CLI命令：

set -e


REQUIREMENTS_FILE=$1
LAYER_HUMAN_NAME=$2
DESCRIPTION=$3
S3_BUCKET=$4


# copying requirements file
mkdir layer -p
cp $REQUIREMENTS_FILE layer/


echo Building...
sam build --use-container -t template.yaml
cd .aws-sam/build/
echo Packaging...
sam package --s3-bucket=$S3_BUCKET -t template.yaml --output-template-file=final_template.yaml
echo Deplopying...
sam deploy -t final_template.yaml --stack-name=layers-$LAYER_HUMAN_NAME --region=us-east-1 --capabilities CAPABILITY_IAM --parameter-overrides "ParameterKey=LayerHumanName,ParameterValue=$LAYER_HUMAN_NAME" "ParameterKey=Description,ParameterValue=\"$DESCRIPTION\""

Pay careful attention to line #13 especially to use-container flag:

请特别注意第13行，尤其是use-container标志：

sam build --use-container ...

From SAM documentation, the use-container option means:

从SAM文档中，use-container选项表示：

If your functions depend on packages that have natively compiled dependencies, use this flag to build your function inside an AWS Lambda-like Docker container.

如果您的函数依赖于具有本机编译依赖性的包，请使用此标志在类似于AWS Lambda的Docker容器中构建函数。

In other words, if you will not turn this flag on, you might end up with incompatible lambda binaries. So, turn it on :)

换句话说，如果您不打开此标志，则可能会导致不兼容的lambda二进制文件。因此，将其打开:)

詹金斯 (Jenkins)

To make the process of Lambda Layer creation as simple as possible we formulated a Jenkins job. A user who wants to create a Layer required to fill this simple Jenkins job form:

为了使Lambda图层的创建过程尽可能简单，我们制定了Jenkins工作。想要创建需要填写此简单Jenkins作业表单的图层的用户：

Code:

码：

properties([
        parameters ([
            choice(choices: ['dev', 'prod'].join("\n"), description: 'What AWS account to use?', name: 'aws_account'),
            string(defaultValue: "", description: 'requirements.txt git repo', name: 'requirements_git_repo'),
            string(defaultValue: "", description: 'requirements.txt git path', name: 'requirements_git_path'),
            string(defaultValue: "", description: 'Layer human readable name, e.g. bijo-tools', name: 'layer_human_name'),
            string(defaultValue: "", description: 'Layer description', name: 'layer_description'),
            string(defaultValue: "my-s3-bucket", description: 's3 bucket for packaging', name: 's3_bucket')


        ])
])




assert params.aws_account != null && params.aws_account.size() != 0                          : "Build failed because aws account name is empty"
assert params.requirements_git_repo != null && params.requirements_git_repo != "" : "Build failed because no requirement.txt git repo was provided"
assert params.requirements_git_path != null && params.requirements_git_path != "" : "Build failed because no requirement.txt git path was provided"
assert params.layer_human_name != null && params.layer_human_name != ""           : "Build failed because no layer humane readable name was provided"
assert params.layer_description != null && params.layer_description != ""                     : "Build failed because no description was provided"
assert params.s3_bucket != ""                                                     : "Build failed because s3 bucket name for packaging was empty"




node {


    checkout scm


    dir("./jenkins/lambda-layers") {
        withAWS(credentials: "aws-${params.aws_account}",region: '<your-aws-region>'){


            withDockerContainer(args: '-v ${PWD}:/tmp/code -w /tmp/code', image: 'lambci/lambda:build-python3.7') {


                String local_repo_path = "${env.WORKSPACE}/requirements_repo/"
                stage('Checkout requirements file from git') {


                    sh("rm -rf ${local_repo_path}")
                    sh("mkdir -p ${local_repo_path}")


                    dir(local_repo_path) {
                         git(url: params.requirements_git_repo, credentialsId: '<YourGitCredentials>', branch: 'master')
                    }
                }


                stage('deploying'){
                    sh "sh requirements_to_layer.sh ${local_repo_path}/${params.requirements_git_path} '${params.layer_human_name}' '${params.layer_description}' '${params.s3_bucket}'"
                }
            }
        }
    }
}

The most important part in this Jenkins job is line #43 which runs our requirements_to_layer.sh:

Jenkins工作中最重要的部分是第43行，该行运行我们的requirements_to_layer.sh：

sh "sh requirements_to_layer.sh ${local_repo_path}/...

However, to enable the smooth execution of this line we first need to have it in Jenkins and use our AWS credentials.

但是，为了使此行能够顺利执行，我们首先需要在Jenkins中使用它并使用我们的AWS凭证。

In lines 1–11, we define the UI parameters needed for this job (the ones you saw in the screenshot). Later, in lines 14–19, we run a few assertions to verify the parameters were entered correctly by the user.

在第1-11行中，我们定义了此作业所需的UI参数(您在屏幕快照中看到的参数)。稍后，在第14-19行中，我们运行一些断言来验证用户是否正确输入了参数。

All 3 code files — sam-lambda-layer.yaml, requirements_to_lambda.sh and lambda_layers.Jenkins stored in a Github repo. We configured our Jenkins job to use that repo. Line #24, ‘checkout scm’ downloads these files from the repo into Jenkins.

所有3个代码文件-sam-lambda-layer.yaml，requirements_to_lambda.sh和lambda_layers.Jenkins存储在Github存储库中。我们将Jenkins作业配置为使用该存储库。第24行，“ checkout scm”将这些文件从存储库下载到Jenkins。

Later, we do this:

稍后，我们这样做：

withAWS(credentials: ..., region: '<your-aws-region>')

‘withAWS’ is using a Jenkins plugin to inject our AWS credentials into this job. ‘sam build', ‘sam package’ and ‘sam deploy’ commands require AWS credentials.

“ withAWS”正在使用Jenkins插件将我们的AWS凭证注入到此作业中。 'sam build'，'sam package'和'sam deploy'命令需要AWS凭证。

Line #29 is not mandatory but recommended, using ‘withDockerContainer’ we run our code inside a Docker container that already contains SAM CLI rather than installing SAM CLI on Jenkins. This is a good practice as it makes it easier to upgrade SAM CLI in the future.

第29行不是强制性的，但建议使用第29行，我们使用“ withDockerContainer”在已经包含SAM CLI的Docker容器中运行代码，而不是在Jenkins上安装SAM CLI。这是一个好习惯，因为这样可以使将来更轻松地升级SAM CLI。

withDockerContainer(..., image: 'lambci/lambda:build-python3.7')

Note: you can also use the official SAM emulation Docker images

注意：您也可以使用官方的SAM仿真Docker映像

概要(Summary)

SAM CLI is simplifying the process of creating Lambda Layers. As an alternative to using manual commands and duplicating shared modules we now use a much easier process.Since SAM is utilizing CloudFormation, we could also manage versions easily and manage our AWS resources in a neater way.

SAM CLI正在简化创建Lambda层的过程。作为使用手动命令和复制共享模块的替代方法，我们现在使用的过程要简单得多。由于SAM使用CloudFormation，我们还可以轻松管理版本并以更整齐的方式管理AWS资源。

To complete the automation picture, we use Jenkins job that simply runs SAM CLI commands for us.

为了完成自动化过程，我们使用Jenkins作业，该作业只为我们运行SAM CLI命令。

This blog post was written in collaboration with Boris Churzin

该博客文章是与Boris Churzin合作撰写的

翻译自: https://medium.com/fundbox-engineering/automating-lambda-layers-with-sam-and-jenkins-53ea8f002c0a

吴雄辉

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
使用Sam和Jenkins自动执行Lambda图层

Our data orchestration platform is using several lambda functions to manage, audit, and monitor data processes. Our Python lambda functions share some logic such as logging, SQS utilities, slack commu...
复制链接

扫一扫