aws lambda 理解RIC和RIE构建和测试容器镜像并通过cdk部署lambda函数

zhaojiew10

已于 2024-05-25 22:36:10 修改

阅读量1.8k

点赞数

分类专栏： AWS 文章标签： aws serverless

于 2023-01-04 00:13:41 首次发布

本文链接：https://blog.csdn.net/sinat_41567654/article/details/128541325

版权

AWS 专栏收录该内容

72 篇文章 4 订阅

订阅专栏

参考资料

lambda容器镜像构建

可以将 Lambda 函数打包和部署最大 10 GB 的容器映像，轻松构建和部署依赖于大量依赖项的更大型工作负载

aws为所有支持的lambda运行时提供了基础镜像，以及基于amazonlinux的自定义镜像。甚至可以自行安装lambda runtime api（RIC运行时接口客户端）对其他的基础镜像（apline或debian）进行扩展

使用lambda容器镜像的先决条件

镜像中安装lambda runtime api
容器必须能够在只读系统上运行，可写的目录（/tmp）大小从512MB到10GB不等
lambda用户有权限运行lambda函数和所需文件
仅支持linux容器，每个镜像针对单一架构

注意：lambda容器镜像仅支持以下dockerfile参数：ENTRYPOINT，CMD，WORKDIR，ENV ，并且会忽略 Dockerfile 中任何不支持的容器镜像设置的值。指定 ENTRYPOINT 或 CMD需要指定绝对路径。

aws发布了Lambda Runtime Interface Emulator（运行时接口模拟器RIE），方便再本地对容器镜像进行测试。预装在基础镜像中。RIE是一个轻量级 Web 服务器，它将 HTTP 请求转换为 JSON 事件以传递给容器镜像中的 Lambda 函数

使用RIE的注意事项

RIE不支持lambda的iam模拟
RIE区分不同架构（x86_64和arm64）
不支持xray或其他lambda集成

将RIE打包到镜像中，以下entry_script.sh脚本通过判断环境变量AWS_LAMBDA_RUNTIME_API决定是否运行RIE

#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
  exec /usr/local/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric $@
else
  exec /usr/local/bin/python -m awslambdaric $@
fi

下载RIE并添加到dockerfile中

COPY ./entry_script.sh /entry_script.sh
ADD aws-lambda-rie-x86_64 /usr/local/bin/aws-lambda-rie
ENTRYPOINT [ "/entry_script.sh" ]

当然也可以不将RIE构建到镜像中，直接在本地测试。将RIE下载安装到本地

mkdir -p ~/.aws-lambda-rie
curl -Lo ~/.aws-lambda-rie/aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie 
chmod +x ~/.aws-lambda-rie/aws-lambda-rie

使用以下命令运行lambda函数

docker run -d -v ~/.aws-lambda-rie:/aws-lambda -p 9000:8080 \
  --entrypoint /aws-lambda/aws-lambda-rie hello-world:latest <image entrypoint> \
      <(optional) image command>

测试发送事件

curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

官方底包构建lambda镜像

aws的基础lambda镜像提供了以下环境变量

LAMBDA_TASK_ROOT=/var/task
LAMBDA_RUNTIME_DIR=/var/runtime

FROM public.ecr.aws/lambda/python:3.8
COPY app.py ${LAMBDA_TASK_ROOT}
COPY requirements.txt  .
RUN  pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
CMD [ "app.handler" ]

备选底包构建lambda镜像

Lambda 支持所有 Linux 发行版，例如 Alpine、Debian 和 Ubuntu，例如以下实例通过多阶段构建安装程序依赖包和ric接口

ARG FUNCTION_DIR="/function"
FROM python:buster as build-image
RUN apt-get update && apt-get install -y g++ make cmake unzip libcurl4-openssl-dev
ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}
COPY app/* ${FUNCTION_DIR}
RUN pip install --target ${FUNCTION_DIR} awslambdaric
# 多阶段构建
FROM python:buster
ARG FUNCTION_DIR
WORKDIR ${FUNCTION_DIR}
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
CMD [ "app.handler" ]

官方的生成PDF文件的简单项目

创建应用程序，通过PDFKit 模块创建pdf文件，使用faker填充随机内容

const PDFDocument = require('pdfkit');
const faker = require('faker');
const getStream = require('get-stream');

exports.lambdaHandler = async (event) => {

    const doc = new PDFDocument();

    const randomName = faker.name.findName();

    doc.text(randomName, { align: 'right' });
    doc.text(faker.address.streetAddress(), { align: 'right' });
    doc.text(faker.address.secondaryAddress(), { align: 'right' });
    doc.text(faker.address.zipCode() + ' ' + faker.address.city(), { align: 'right' });
    doc.moveDown();
    doc.text('Dear ' + randomName + ',');
    doc.moveDown();
    for(let i = 0; i < 3; i++) {
        doc.text(faker.lorem.paragraph());
        doc.moveDown();
    }
    doc.text(faker.name.findName(), { align: 'right' });
    doc.end();

    pdfBuffer = await getStream.buffer(doc);
    pdfBase64 = pdfBuffer.toString('base64');

    const response = {
        statusCode: 200,
        headers: {
            'Content-Length': Buffer.byteLength(pdfBase64),
            'Content-Type': 'application/pdf',
            'Content-disposition': 'attachment;filename=test.pdf'
        },
        isBase64Encoded: true,
        body: pdfBase64
    };
    return response;
};

生成依赖文件package.json

npm init -y
npm install pdfkit
npm install faker
npm install get-stream

创建dockerfile，CMD中指定lambda入口

FROM amazon/aws-lambda-nodejs:16
COPY app.js package*.json ./
RUN npm install
CMD [ "app.lambdaHandler" ]

构建容器

docker build -t random-letter .

运行测试，响应成功，可见amazon镜像自带RIE和RIC

docker run -p 9000:8080 random-letter:latest
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
output:
{"statusCode":200,"headers":{"Content-Length":2572,"Content-Type":"application/pdf","Content-disposition":"attachment;filename=test.pdf"},"isBase64Encoded":true,"body":"JVBERi0xLjMKJf//....."}

将构建好的容器上传到ecr中，略去

创建lambda函数并选择容器镜像即可

在这里插入图片描述

通过这个脚本将bese64结果还原成pdf，可见函数创建成功

在这里插入图片描述

基于自定义镜像影响构建lambda镜像

参考：https://aws.amazon.com/cn/blogs/china/new-for-aws-lambda-container-image-support/

初始化项目

pipenv --python 3.9
pipenv install faker
pipenv requirements > requirements.txt

创建一个简单的app.py，用faker造一些数据

import sys
from faker import Faker
def handler(event, context): 
    fake = Faker()
    play = {"name":fake.name(),"address":fake.address,"dialogue":fake.text()}
    print(play)
    return 'Hello from AWS Lambda using Python' + sys.version + '!'

创建入口文件entry.sh，这个之前解释过了.

注意：中文blog上没有$1，以后翻译的东西还是别看了，符号都丢

#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
    exec /usr/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric $1
else
    exec /usr/local/bin/python -m awslambdaric $1
fi

创建dockerfile，注意：在第2阶段需要使用gcc编译和关联ric的依赖项

ARG FUNCTION_DIR="/home/app/"
ARG RUNTIME_VERSION="3.9"
ARG DISTRO_VERSION="3.12"
#stage 1
FROM python:${RUNTIME_VERSION}-alpine${DISTRO_VERSION} AS python-alpine
RUN apk add --no-cache libstdc++
#stage 2
FROM python-alpine AS build-image
RUN apk add --no-cache build-base libtool autoconf automake libexecinfo-dev make cmake libcurl
ARG FUNCTION_DIR
ARG RUNTIME_VERSION
RUN mkdir -p ${FUNCTION_DIR}
COPY ./* ${FUNCTION_DIR}
RUN ls ${FUNCTION_DIR}
RUN python${RUNTIME_VERSION} -m pip install -r ${FUNCTION_DIR}/requirements.txt --target ${FUNCTION_DIR}
RUN python${RUNTIME_VERSION} -m pip install awslambdaric --target ${FUNCTION_DIR} -i https://pypi.douban.com/simple/
#stage 3
FROM python-alpine
ARG FUNCTION_DIR
WORKDIR ${FUNCTION_DIR}
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/bin/aws-lambda-rie
RUN chmod 755 /usr/bin/aws-lambda-rie
COPY entry.sh /
# ENTRYPOINT [ "/entry.sh" ]
ENTRYPOINT [ "/usr/local/bin/python", “-m”, “awslambdaric” ]
CMD [ "app.handler" ]

手动上传image到ecr中创建lambda函数之后，测试通过

aws lambda invoke --cli-binary-format raw-in-base64-out --function-name sampleContainerFunction  --payload '{ "name": "Bob" }' response.json

在这里插入图片描述

本地测试

docker build -t sayhello .
docker run -p 9000:8080 sayhello
docker run -it --rm -p 9000:8080 sayhello
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

由于RIE是可选的，我们可以将RIE安装在本地进行测试

注释dockerfile中的RIE，修改ENTRYPOINT 为 ENTRYPOINT [ "/usr/local/bin/python", “-m”, “awslambdaric” ]，运行以下命令测试

docker run -d -v ~/.aws-lambda-rie:/aws-lambda -p 9000:8080 \
       --entrypoint /aws-lambda/aws-lambda-rie sayhello
       /lambda-entrypoint.sh app.handler

遇到的错误

（1）在本地运行时出现以下报错

KeyError: 'AWS_LAMBDA_RUNTIME_API'

当函数部署到lambda服务时，运行时会从 AWS_LAMBDA_RUNTIME_API 环境变量获取 API 终端节点，本地测试无法获取AWS_LAMBDA_RUNTIME_API环境变量所以报错，需要使用RIE入口的entry.sh构建

https://docs.amazonaws.cn/lambda/latest/dg/runtimes-api.html

（2）在本地通过RIE请求测试出现以下报错

Traceback (most recent call last):
  File "/home/app/awslambdaric/__main__.py", line 15, in main
    handler = args[1]
IndexError: list index out of range

检查发现入口文件可能有点问题，修改如下

#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
    exec /usr/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric $@
else
    exec /usr/local/bin/python -m awslambdaric $@
fi

通过cdk打包上传oci镜像创建lambda函数

https://docs.aws.amazon.com/cdk/api/v2/python/index.html

lambda-from-container-demo

初始化cdk项目（python）

cdk init --language=python

复制示例项目中的堆栈代码，略作修改

#!/usr/bin/env python3
import os
import typing
import aws_cdk as cdk
from aws_cdk import (
    aws_lambda,
    aws_ecr,
    Aws, Duration, Stack, Construct
)
# from lambdafromcontainer.lambdafromcontainer_stack import LambdafromcontainerStack

class LambdaContainerFunctionStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)
        image_name    = "lambdaContainerFunction"
        # if use pre_existing image
        use_pre_existing_image = False
        if (use_pre_existing_image):
            ecr_repository = aws_ecr.Repository.from_repository_attributes(self,
                id              = "ECR",
                repository_name = image_name
            )
            
            ecr_image = typing.cast("aws_lambda.Code", aws_lambda.EcrImageCode(
                repository = ecr_repository
            ))
        else:
            ecr_image = aws_lambda.EcrImageCode.from_asset_image(
                directory = os.path.join(os.getcwd(), "lambda-image")
            )

        aws_lambda.Function(self,
            id            = "lambdaContainerFunction",
            description   = "Sample Lambda Container Function",
            code          = ecr_image,
            handler       = aws_lambda.Handler.FROM_IMAGE,
            runtime       = aws_lambda.Runtime.FROM_IMAGE,
            environment   = {"hello":"world"},
            function_name = "sampleContainerFunction",
            memory_size   = 128,
            reserved_concurrent_executions = 10,
            timeout       = Duration.seconds(10),
        )

app = cdk.App()
LambdaContainerFunctionStack(app, "LambdafromcontainerStack",
    env=cdk.Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'), region=os.getenv('CDK_DEFAULT_REGION')),
    )
app.synth()

通过directory = os.path.join(os.getcwd(), "lambda-image")

项目结构有，app还是用之前的pdf应用

$ tree 
├── app.py
├── cdk.json
├── lambda-image
│   ├── app.js
│   ├── Dockerfile
│   └── package.json
├── README.md
├── requirements-dev.txt
├── requirements.txt
├── source.bat
└── tests
    ├── __init__.py
    └── unit
        ├── __init__.py
        └── test_lambdafromcontainer_stack.py

最后生成的cfn模板如下

Resources:
  lambdaContainerFunctionServiceRole5E36DB3C:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
  lambdaContainerFunction5815FD88:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ImageUri:
          Fn::Sub: xxxxxxxxx.dkr.ecr.cn-north-1.${AWS::URLSuffix}/cdk-hnb659fds-container-assets-xxxxxxxxx-cn-north-1:64c8e7a88147f47764a7d9704e172e6ee65c807bc0157fac194da92f82afadcd
      Role:
        Fn::GetAtt:
          - lambdaContainerFunctionServiceRole5E36DB3C
          - Arn
      Description: Sample Lambda Container Function
      Environment:
        Variables:
          hello: world
      FunctionName: sampleContainerFunction
      MemorySize: 128
      PackageType: Image
      ReservedConcurrentExecutions: 10
      Timeout: 10
    DependsOn:
      - lambdaContainerFunctionServiceRole5E36DB3C