参考资料
lambda容器镜像构建
可以将 Lambda 函数打包和部署最大 10 GB 的容器映像,轻松构建和部署依赖于大量依赖项的更大型工作负载
aws为所有支持的lambda运行时提供了基础镜像,以及基于amazonlinux的自定义镜像。甚至可以自行安装lambda runtime api(RIC运行时接口客户端)对其他的基础镜像(apline或debian)进行扩展
使用lambda容器镜像的先决条件
- 镜像中安装lambda runtime api
- 容器必须能够在只读系统上运行,可写的目录(/tmp)大小从512MB到10GB不等
- lambda用户有权限运行lambda函数和所需文件
- 仅支持linux容器,每个镜像针对单一架构
注意:lambda容器镜像仅支持以下dockerfile参数:ENTRYPOINT,CMD,WORKDIR,ENV ,并且会忽略 Dockerfile 中任何不支持的容器镜像设置的值。指定 ENTRYPOINT 或 CMD需要指定绝对路径。
aws发布了Lambda Runtime Interface Emulator(运行时接口模拟器RIE),方便再本地对容器镜像进行测试。预装在基础镜像中。RIE是一个轻量级 Web 服务器,它将 HTTP 请求转换为 JSON 事件以传递给容器镜像中的 Lambda 函数
使用RIE的注意事项
- RIE不支持lambda的iam模拟
- RIE区分不同架构(x86_64和arm64)
- 不支持xray或其他lambda集成
将RIE打包到镜像中,以下entry_script.sh
脚本通过判断环境变量AWS_LAMBDA_RUNTIME_API
决定是否运行RIE
#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
exec /usr/local/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric $@
else
exec /usr/local/bin/python -m awslambdaric $@
fi
下载RIE并添加到dockerfile中
COPY ./entry_script.sh /entry_script.sh
ADD aws-lambda-rie-x86_64 /usr/local/bin/aws-lambda-rie
ENTRYPOINT [ "/entry_script.sh" ]
当然也可以不将RIE构建到镜像中,直接在本地测试。将RIE下载安装到本地
mkdir -p ~/.aws-lambda-rie
curl -Lo ~/.aws-lambda-rie/aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie
chmod +x ~/.aws-lambda-rie/aws-lambda-rie
使用以下命令运行lambda函数
docker run -d -v ~/.aws-lambda-rie:/aws-lambda -p 9000:8080 \
--entrypoint /aws-lambda/aws-lambda-rie hello-world:latest <image entrypoint> \
<(optional) image command>
测试发送事件
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
官方底包构建lambda镜像
aws的基础lambda镜像提供了以下环境变量
LAMBDA_TASK_ROOT=/var/task
LAMBDA_RUNTIME_DIR=/var/runtime
FROM public.ecr.aws/lambda/python:3.8
COPY app.py ${LAMBDA_TASK_ROOT}
COPY requirements.txt .
RUN pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
CMD [ "app.handler" ]
备选底包构建lambda镜像
Lambda 支持所有 Linux 发行版,例如 Alpine、Debian 和 Ubuntu,例如以下实例通过多阶段构建安装程序依赖包和ric接口
ARG FUNCTION_DIR="/function"
FROM python:buster as build-image
RUN apt-get update && apt-get install -y g++ make cmake unzip libcurl4-openssl-dev
ARG FUNCTION_DIR
RUN mkdir -p ${FUNCTION_DIR}
COPY app/* ${FUNCTION_DIR}
RUN pip install --target ${FUNCTION_DIR} awslambdaric
# 多阶段构建
FROM python:buster
ARG FUNCTION_DIR
WORKDIR ${FUNCTION_DIR}
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
CMD [ "app.handler" ]
官方的生成PDF文件的简单项目
创建应用程序,通过PDFKit 模块创建pdf文件,使用faker填充随机内容
const PDFDocument = require('pdfkit');
const faker = require('faker');
const getStream = require('get-stream');
exports.lambdaHandler = async (event) => {
const doc = new PDFDocument();
const randomName = faker.name.findName();
doc.text(randomName, { align: 'right' });
doc.text(faker.address.streetAddress(), { align: 'right' });
doc.text(faker.address.secondaryAddress(), { align: 'right' });
doc.text(faker.address.zipCode() + ' ' + faker.address.city(), { align: 'right' });
doc.moveDown();
doc.text('Dear ' + randomName + ',');
doc.moveDown();
for(let i = 0; i < 3; i++) {
doc.text(faker.lorem.paragraph());
doc.moveDown();
}
doc.text(faker.name.findName(), { align: 'right' });
doc.end();
pdfBuffer = await getStream.buffer(doc);
pdfBase64 = pdfBuffer.toString('base64');
const response = {
statusCode: 200,
headers: {
'Content-Length': Buffer.byteLength(pdfBase64),
'Content-Type': 'application/pdf',
'Content-disposition': 'attachment;filename=test.pdf'
},
isBase64Encoded: true,
body: pdfBase64
};
return response;
};
生成依赖文件package.json
npm init -y
npm install pdfkit
npm install faker
npm install get-stream
创建dockerfile,CMD中指定lambda入口
FROM amazon/aws-lambda-nodejs:16
COPY app.js package*.json ./
RUN npm install
CMD [ "app.lambdaHandler" ]
构建容器
docker build -t random-letter .
运行测试,响应成功,可见amazon镜像自带RIE和RIC
docker run -p 9000:8080 random-letter:latest
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
output:
{"statusCode":200,"headers":{"Content-Length":2572,"Content-Type":"application/pdf","Content-disposition":"attachment;filename=test.pdf"},"isBase64Encoded":true,"body":"JVBERi0xLjMKJf//....."}
将构建好的容器上传到ecr中,略去
创建lambda函数并选择容器镜像即可
通过这个脚本将bese64结果还原成pdf,可见函数创建成功
基于自定义镜像影响构建lambda镜像
参考:https://aws.amazon.com/cn/blogs/china/new-for-aws-lambda-container-image-support/
初始化项目
pipenv --python 3.9
pipenv install faker
pipenv requirements > requirements.txt
创建一个简单的app.py,用faker造一些数据
import sys
from faker import Faker
def handler(event, context):
fake = Faker()
play = {"name":fake.name(),"address":fake.address,"dialogue":fake.text()}
print(play)
return 'Hello from AWS Lambda using Python' + sys.version + '!'
创建入口文件entry.sh
,这个之前解释过了.
注意:中文blog上没有$1
,以后翻译的东西还是别看了,符号都丢
#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
exec /usr/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric $1
else
exec /usr/local/bin/python -m awslambdaric $1
fi
创建dockerfile,注意:在第2阶段需要使用gcc编译和关联ric的依赖项
ARG FUNCTION_DIR="/home/app/"
ARG RUNTIME_VERSION="3.9"
ARG DISTRO_VERSION="3.12"
#stage 1
FROM python:${RUNTIME_VERSION}-alpine${DISTRO_VERSION} AS python-alpine
RUN apk add --no-cache libstdc++
#stage 2
FROM python-alpine AS build-image
RUN apk add --no-cache build-base libtool autoconf automake libexecinfo-dev make cmake libcurl
ARG FUNCTION_DIR
ARG RUNTIME_VERSION
RUN mkdir -p ${FUNCTION_DIR}
COPY ./* ${FUNCTION_DIR}
RUN ls ${FUNCTION_DIR}
RUN python${RUNTIME_VERSION} -m pip install -r ${FUNCTION_DIR}/requirements.txt --target ${FUNCTION_DIR}
RUN python${RUNTIME_VERSION} -m pip install awslambdaric --target ${FUNCTION_DIR} -i https://pypi.douban.com/simple/
#stage 3
FROM python-alpine
ARG FUNCTION_DIR
WORKDIR ${FUNCTION_DIR}
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/bin/aws-lambda-rie
RUN chmod 755 /usr/bin/aws-lambda-rie
COPY entry.sh /
# ENTRYPOINT [ "/entry.sh" ]
ENTRYPOINT [ "/usr/local/bin/python", “-m”, “awslambdaric” ]
CMD [ "app.handler" ]
手动上传image到ecr中创建lambda函数之后,测试通过
aws lambda invoke --cli-binary-format raw-in-base64-out --function-name sampleContainerFunction --payload '{ "name": "Bob" }' response.json
本地测试
docker build -t sayhello .
docker run -p 9000:8080 sayhello
docker run -it --rm -p 9000:8080 sayhello
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
由于RIE是可选的,我们可以将RIE安装在本地进行测试
注释dockerfile中的RIE,修改ENTRYPOINT
为 ENTRYPOINT [ "/usr/local/bin/python", “-m”, “awslambdaric” ]
,运行以下命令测试
docker run -d -v ~/.aws-lambda-rie:/aws-lambda -p 9000:8080 \
--entrypoint /aws-lambda/aws-lambda-rie sayhello
/lambda-entrypoint.sh app.handler
遇到的错误
(1)在本地运行时出现以下报错
KeyError: 'AWS_LAMBDA_RUNTIME_API'
当函数部署到lambda服务时,运行时会从 AWS_LAMBDA_RUNTIME_API
环境变量获取 API 终端节点,本地测试无法获取AWS_LAMBDA_RUNTIME_API
环境变量所以报错,需要使用RIE入口的entry.sh
构建
https://docs.amazonaws.cn/lambda/latest/dg/runtimes-api.html
(2)在本地通过RIE请求测试出现以下报错
Traceback (most recent call last):
File "/home/app/awslambdaric/__main__.py", line 15, in main
handler = args[1]
IndexError: list index out of range
检查发现入口文件可能有点问题,修改如下
#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
exec /usr/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric $@
else
exec /usr/local/bin/python -m awslambdaric $@
fi
通过cdk打包上传oci镜像创建lambda函数
https://docs.aws.amazon.com/cdk/api/v2/python/index.html
初始化cdk项目(python)
cdk init --language=python
复制示例项目中的堆栈代码,略作修改
#!/usr/bin/env python3
import os
import typing
import aws_cdk as cdk
from aws_cdk import (
aws_lambda,
aws_ecr,
Aws, Duration, Stack, Construct
)
# from lambdafromcontainer.lambdafromcontainer_stack import LambdafromcontainerStack
class LambdaContainerFunctionStack(Stack):
def __init__(self, scope: Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
image_name = "lambdaContainerFunction"
# if use pre_existing image
use_pre_existing_image = False
if (use_pre_existing_image):
ecr_repository = aws_ecr.Repository.from_repository_attributes(self,
id = "ECR",
repository_name = image_name
)
ecr_image = typing.cast("aws_lambda.Code", aws_lambda.EcrImageCode(
repository = ecr_repository
))
else:
ecr_image = aws_lambda.EcrImageCode.from_asset_image(
directory = os.path.join(os.getcwd(), "lambda-image")
)
aws_lambda.Function(self,
id = "lambdaContainerFunction",
description = "Sample Lambda Container Function",
code = ecr_image,
handler = aws_lambda.Handler.FROM_IMAGE,
runtime = aws_lambda.Runtime.FROM_IMAGE,
environment = {"hello":"world"},
function_name = "sampleContainerFunction",
memory_size = 128,
reserved_concurrent_executions = 10,
timeout = Duration.seconds(10),
)
app = cdk.App()
LambdaContainerFunctionStack(app, "LambdafromcontainerStack",
env=cdk.Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'), region=os.getenv('CDK_DEFAULT_REGION')),
)
app.synth()
通过directory = os.path.join(os.getcwd(), "lambda-image")
项目结构有,app还是用之前的pdf应用
$ tree
├── app.py
├── cdk.json
├── lambda-image
│ ├── app.js
│ ├── Dockerfile
│ └── package.json
├── README.md
├── requirements-dev.txt
├── requirements.txt
├── source.bat
└── tests
├── __init__.py
└── unit
├── __init__.py
└── test_lambdafromcontainer_stack.py
最后生成的cfn模板如下
Resources:
lambdaContainerFunctionServiceRole5E36DB3C:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: "2012-10-17"
ManagedPolicyArns:
- Fn::Join:
- ""
- - "arn:"
- Ref: AWS::Partition
- :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
lambdaContainerFunction5815FD88:
Type: AWS::Lambda::Function
Properties:
Code:
ImageUri:
Fn::Sub: xxxxxxxxx.dkr.ecr.cn-north-1.${AWS::URLSuffix}/cdk-hnb659fds-container-assets-xxxxxxxxx-cn-north-1:64c8e7a88147f47764a7d9704e172e6ee65c807bc0157fac194da92f82afadcd
Role:
Fn::GetAtt:
- lambdaContainerFunctionServiceRole5E36DB3C
- Arn
Description: Sample Lambda Container Function
Environment:
Variables:
hello: world
FunctionName: sampleContainerFunction
MemorySize: 128
PackageType: Image
ReservedConcurrentExecutions: 10
Timeout: 10
DependsOn:
- lambdaContainerFunctionServiceRole5E36DB3C
注意,lambda-image目录下的dockerfile的名称必须首字母大写Dockerfile
直接部署lambda函数
cdk deploy
以上命令会自动构建镜像并以该镜像创建lambda函数,减少了很多工作量
在lambda控制台测试成功