【Paddle2ONNX】为Paddle2ONNX搭建自动发包机制

最新推荐文章于 2024-12-17 13:07:29 发布

ZhengBicheng0522

最新推荐文章于 2024-12-17 13:07:29 发布

阅读量2.2k

点赞数 55

分类专栏： Paddle开源社区文章标签：人工智能 paddlepaddle 开源软件深度学习

本文链接：https://blog.csdn.net/qq_37380933/article/details/137712015

版权

Paddle开源社区专栏收录该内容

13 篇文章

订阅专栏

本文介绍了如何为Paddle2ONNX项目引入自动发包机制，通过构建脚本编译protobuf和不同Python版本的wheel文件，并利用GitHubActions和Docker在多个架构上进行编译，最终通过PyPI发布。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

简介

Paddle2ONNX此前一直使用一个一个的编译所有版本的Python源码包再手动上传到PyPI的方式来分发发行版。很显然，这是一种极其低效的办法，本文介绍如何为Paddle2ONNX添加自动发包机制。

实现过程

Paddle2ONNX的编译流程参考onnx搭建，因此在设计编译流程上我也参考了ONNX。

新建编译protobuf脚本

首先新建一个脚本用于编译protobuf

#!/bin/bash

# Copyright (c) ONNX Project Contributors
#
# SPDX-License-Identifier: Apache-2.0

export CORE_NUMBER=$1

if [[ -z "$CORE_NUMBER" ]]; then
   export CORE_NUMBER=1
fi

# Build protobuf from source with -fPIC on Unix-like system
ORIGINAL_PATH=$(pwd)
cd ..
git clone https://github.com/protocolbuffers/protobuf.git -b v3.16.0
cd protobuf
mkdir build_source && cd build_source
cmake ../cmake -DCMAKE_INSTALL_PREFIX=${PWD}/installed_protobuf_lib -Dprotobuf_BUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE=Release
make -j$CORE_NUMBER && make install
export PATH=${PWD}/installed_protobuf_lib/bin:${PATH}
cd $ORIGINAL_PATH

新建编译Paddle2ONNX脚本

然后新建一个脚本传入参数编译Paddle2ONNX

#!/bin/bash

# Copyright (c) ONNX Project Contributors
#
# SPDX-License-Identifier: Apache-2.0

set -e -x

# CLI arguments
PY_VERSION=$1
PLAT=$2
GITHUB_EVENT_NAME=$3

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib

# Compile wheels
# Need to be updated if there is a new Python Version
if [ "$(uname -m)" == "aarch64" ]; then
 PIP_INSTALL_COMMAND="$PY_VERSION -m pip install --no-cache-dir -q"
 PYTHON_COMMAND="$PY_VERSION"
else
 declare -A python_map=( ["3.8"]="cp38-cp38" ["3.9"]="cp39-cp39" ["3.10"]="cp310-cp310" ["3.11"]="cp311-cp311" ["3.12"]="cp312-cp312")
 PY_VER=${python_map[$PY_VERSION]}
 PIP_INSTALL_COMMAND="/opt/python/${PY_VER}/bin/pip install --no-cache-dir -q"
 PYTHON_COMMAND="/opt/python/${PY_VER}/bin/python"
fi

# Update pip and install cmake
$PIP_INSTALL_COMMAND --upgrade pip
$PIP_INSTALL_COMMAND cmake

# Build protobuf from source
source .github/workflows/scripts/build_protobuf_unix.sh "$(nproc)"

# Build Paddle2ONNX wheels
$PYTHON_COMMAND -m build --wheel || { echo "Building wheels failed."; exit 1; }

# Bundle external shared libraries into the wheels
# find -exec does not preserve failed exit codes, so use an output file for failures
failed_wheels=$PWD/failed-wheels
rm -f "$failed_wheels"
find . -type f -iname "*-linux*.whl" -exec sh -c "auditwheel repair '{}' -w \$(dirname '{}') --plat '${PLAT}' || { echo 'Repairing wheels failed.'; auditwheel show '{}' >> '$failed_wheels'; }" \;

if [[ -f "$failed_wheels" ]]; then
    echo "Repairing wheels failed:"
    cat failed-wheels
    exit 1
fi

# Remove useless *-linux*.whl; only keep manylinux*.whl
rm -f dist/*-linux*.whl

echo "Successfully build wheels:"
find . -type f -iname "*manylinux*.whl"

新建发包工作流

我们需要同时编译多个版本的Python包，对运行系统的配置如下:

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [ '3.8', '3.9', '3.10', '3.11', '3.12']
        architecture: [ 'x64' ]

接着需要检出Paddle2ONNX的最新分支

jobs:
  build:
    steps:
      # Checkout the latest branch of Paddle2ONNX.
      - name: Checkout Paddle2ONNX
        uses: actions/checkout@v4
        with:
          submodules: true

为了保证编译出来的源码包能在更多系统上使用，我们需要使用docker进行编译

jobs:
  build:
    steps:
      - name: Build on manylinux2014_x86_64
        uses: docker://quay.io/pypa/manylinux2014_x86_64:latest
        with:
          entrypoint: bash
          args: .github/workflows/scripts/entrypoint.sh ${{ matrix.python-version }} manylinux2014_x86_64

最后还需要对自动发包进行配置

jobs:
  build:
    steps:
      - uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
        with:
          name: wheels
          path: dist

      - name: Publish package
        uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
        with:
          user: __token__
          password: ${{ secrets.ZHENG_BICHENG_PYPI_TOKEN }}

整体文件构造如下:

name: Build Package

on:
  release:
    types: [published]

permissions:
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [ '3.8', '3.9', '3.10', '3.11', '3.12']
        architecture: [ 'x64' ]

    # For the sake of simplicity in testing, the Paddle2ONNX packaging program will temporarily only package executable files for Python 3.8.
    # In the future, we will need to extend support to cover Python 3.8 through Python 3.10.
    steps:
      # Checkout the latest branch of Paddle2ONNX.
      - name: Checkout Paddle2ONNX
        uses: actions/checkout@v4
        with:
          submodules: true

      - name: Build on manylinux2014_x86_64
        uses: docker://quay.io/pypa/manylinux2014_x86_64:latest
        with:
          entrypoint: bash
          args: .github/workflows/scripts/entrypoint.sh ${{ matrix.python-version }} manylinux2014_x86_64

      - uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
        with:
          name: wheels
          path: dist

      - name: Publish package
        uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
        with:
          user: __token__
          password: ${{ secrets.ZHENG_BICHENG_PYPI_TOKEN }}