使用setup.py打包 HuggingFace PEFT 项目详解：pip install peft的幕后流程-CSDN博客

本文链接：https://blog.csdn.net/shizheng_Li/article/details/144832584

使用 `setup.py` 打包 HuggingFace PEFT 项目详解

Source: https://github.com/huggingface/peft/blob/main/setup.py

1. 项目简介

HuggingFace 的 PEFT（Parameter-Efficient Fine-Tuning）库是一个用于高效参数微调的 Python 工具包，支持多种大语言模型（LLM）微调方法。本文将详细解析其 setup.py 文件，并说明如何打包和发布该项目到 PyPI。

2. 打包的核心依赖文件解析

PEFT 项目采用 setuptools 进行打包配置，核心依赖文件包括：

setup.py：配置项目名称、版本号、依赖项等关键信息。
LICENSE：项目许可证说明，Apache 2.0。
README.md：项目详细描述，作为 PyPI 上的主页内容。
src 文件夹：包含源代码，代码结构清晰。
find_packages()：自动寻找 src 目录中的 Python 包，简化手动定义路径的复杂度。

3. 关键代码解析

3.1 基本信息

VERSION = "0.14.1.dev0"  # 当前版本号

版本号定义为 0.14.1 开发版，后续发布新版本需要更新此处。

setup(
    name="peft",  # 项目名称
    version=VERSION,  # 版本号
    description="Parameter-Efficient Fine-Tuning (PEFT)",  # 项目描述
    license_files=["LICENSE"],  # 许可证文件
    long_description=open("README.md", encoding="utf-8").read(),  # 详细说明
    long_description_content_type="text/markdown",  # 描述内容格式
    keywords="deep learning",  # 关键词，方便搜索
    license="Apache",  # 许可证类型
    author="The HuggingFace team",  # 作者
    author_email="benjamin@huggingface.co",  # 联系邮箱
    url="https://github.com/huggingface/peft",  # 项目主页地址

上述代码提供了基础项目信息，这些信息会直接显示在 PyPI 上。

3.2 安装依赖项

install_requires=[
    "numpy>=1.17",
    "packaging>=20.0",
    "psutil",
    "pyyaml",
    "torch>=1.13.0",
    "transformers",
    "tqdm",
    "accelerate>=0.21.0",
    "safetensors",
    "huggingface_hub>=0.25.0",
],

核心依赖：
- numpy：矩阵和数据处理。
- torch：深度学习框架。
- transformers：模型加载与微调。
- accelerate：优化并行训练与推理。
- safetensors：安全存储模型参数。
- huggingface_hub：集成 HuggingFace 模型和工具。

3.3 扩展依赖项

extras = {}
extras["quality"] = [
    "black",  # doc-builder has an implicit dependency on Black, see huggingface/doc-builder#434
    "hf-doc-builder",
    "ruff~=0.6.1",
]
extras["docs_specific"] = [
    "black",  # doc-builder has an implicit dependency on Black, see huggingface/doc-builder#434
    "hf-doc-builder",
]
extras["dev"] = extras["quality"] + extras["docs_specific"]
extras["test"] = extras["dev"] + [
    "pytest",
    "pytest-cov",
    "pytest-xdist",
    "parameterized",
    "datasets",
    "diffusers",
    "scipy",
    "protobuf",
    "sentencepiece",
]

扩展功能分类：
- quality：代码格式检查和文档生成。
- dev：开发和测试环境的扩展依赖。
- test：单元测试与代码覆盖率工具。

这些扩展依赖支持开发者在不同需求下安装特定工具集。例如，开发者可以使用以下命令安装开发环境：

pip install .[dev]

3.4 包管理与源代码组织

package_dir={"": "src"},
packages=find_packages("src"),
package_data={"peft": ["py.typed", "tuners/boft/fbd/fbd_cuda.cpp", "tuners/boft/fbd/fbd_cuda_kernel.cu"]},

package_dir：指定源代码位于 src 目录下。
find_packages()：自动检索并打包所有子模块。
package_data：明确包含的额外资源文件，如 .cpp 和 .cu 文件。

3.5 分类标签

classifiers=[
    "Development Status :: 5 - Production/Stable",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: Apache Software License",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3",
    "Topic :: Scientific/Engineering :: Artificial Intelligence",
],

这些标签描述项目的开发状态、适用人群和环境，为 PyPI 用户提供筛选条件。例如：

开发状态：5 - Production/Stable 表示稳定版。
操作系统：支持所有平台。

4. 打包与发布过程

4.1 构建包

python setup.py sdist bdist_wheel

sdist：生成源码包，适用于不同平台的源码安装。
bdist_wheel：生成二进制包，加速安装过程。

生成的包将保存在 dist/ 目录下。

4.2 上传测试 PyPI

twine upload dist/* -r pypitest

4.3 验证安装

pip install -i https://testpypi.python.org/pypi --extra-index-url https://pypi.org/simple peft

4.4 上传正式 PyPI

twine upload dist/* -r pypi

5. 版本管理与更新

版本更新步骤

更新版本号：
```
VERSION = "0.14.2.dev0"
```

提交代码并创建新标签：

git tag -a 0.14.2 -m "Release version 0.14.2"
git push --tags origin main

发布包到 PyPI：

python setup.py sdist bdist_wheel
twine upload dist/*

对应的原文：

# Release checklist
# 1. Change the version in __init__.py and setup.py to the release version, e.g. from "0.6.0.dev0" to "0.6.0"
# 2. Check if there are any deprecations that need to be addressed for this release by searching for "# TODO" in the code
# 3. Commit these changes with the message: "Release: VERSION", create a PR and merge it.
# 4. Add a tag in git to mark the release: "git tag -a VERSION -m 'Adds tag VERSION for pypi' "
#    Push the tag to git:
#      git push --tags origin main
#    It is necessary to work on the original repository, not on a fork.
# 5. Run the following commands in the top-level directory:
#      python setup.py bdist_wheel
#      python setup.py sdist
#    Ensure that you are on the clean and up-to-date main branch (git status --untracked-files=no should not list any
#    files and show the main branch)
# 6. Upload the package to the pypi test server first:
#      twine upload dist/* -r pypitest
# 7. Check that you can install it in a virtualenv by running:
#      pip install -i https://testpypi.python.org/pypi --extra-index-url https://pypi.org/simple peft
# 8. Upload the final version to actual pypi:
#      twine upload dist/* -r pypi
# 9. Add release notes to the tag on https://github.com/huggingface/peft/releases once everything is looking hunky-dory.
#      Check the notes here: https://docs.google.com/document/d/1k-sOIfykuKjWcOIALqjhFKz4amFEp-myeJUJEzNgjoU/edit?usp=sharing
# 10. Update the version in __init__.py, setup.py to the bumped minor version + ".dev0" (e.g. from "0.6.0" to "0.7.0.dev0")

6. 小结

通过上述分析可以看出，PEFT 项目的打包流程十分标准化，主要依赖 setuptools 和 twine 工具。开发者可以根据不同需求扩展依赖项，并轻松实现版本管理与发布。

PEFT 的 setup.py 设计符合 Python 打包规范，支持源码和二进制包的分发，适合需要快速部署 AI 模型微调工具的用户参考和使用。

后记

2024年12月30日19点23分于上海，在GPT4o大大模型辅助下完成。

附录

https://github.com/huggingface/peft/blob/main/setup.py源代码如下：可以结合源代码阅读本文。

# Copyright 2023 The HuggingFace Team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from setuptools import find_packages, setup


VERSION = "0.14.1.dev0"

extras = {}
extras["quality"] = [
    "black",  # doc-builder has an implicit dependency on Black, see huggingface/doc-builder#434
    "hf-doc-builder",
    "ruff~=0.6.1",
]
extras["docs_specific"] = [
    "black",  # doc-builder has an implicit dependency on Black, see huggingface/doc-builder#434
    "hf-doc-builder",
]
extras["dev"] = extras["quality"] + extras["docs_specific"]
extras["test"] = extras["dev"] + [
    "pytest",
    "pytest-cov",
    "pytest-xdist",
    "parameterized",
    "datasets",
    "diffusers",
    "scipy",
    "protobuf",
    "sentencepiece",
]

setup(
    name="peft",
    version=VERSION,
    description="Parameter-Efficient Fine-Tuning (PEFT)",
    license_files=["LICENSE"],
    long_description=open("README.md", encoding="utf-8").read(),
    long_description_content_type="text/markdown",
    keywords="deep learning",
    license="Apache",
    author="The HuggingFace team",
    author_email="benjamin@huggingface.co",
    url="https://github.com/huggingface/peft",
    package_dir={"": "src"},
    packages=find_packages("src"),
    package_data={"peft": ["py.typed", "tuners/boft/fbd/fbd_cuda.cpp", "tuners/boft/fbd/fbd_cuda_kernel.cu"]},
    entry_points={},
    python_requires=">=3.9.0",
    install_requires=[
        "numpy>=1.17",
        "packaging>=20.0",
        "psutil",
        "pyyaml",
        "torch>=1.13.0",
        "transformers",
        "tqdm",
        "accelerate>=0.21.0",
        "safetensors",
        "huggingface_hub>=0.25.0",
    ],
    extras_require=extras,
    classifiers=[
        "Development Status :: 5 - Production/Stable",
        "Intended Audience :: Developers",
        "Intended Audience :: Education",
        "Intended Audience :: Science/Research",
        "License :: OSI Approved :: Apache Software License",
        "Operating System :: OS Independent",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.9",
        "Programming Language :: Python :: 3.10",
        "Programming Language :: Python :: 3.11",
        "Programming Language :: Python :: 3.12",
        "Topic :: Scientific/Engineering :: Artificial Intelligence",
    ],
)

# Release checklist
# 1. Change the version in __init__.py and setup.py to the release version, e.g. from "0.6.0.dev0" to "0.6.0"
# 2. Check if there are any deprecations that need to be addressed for this release by searching for "# TODO" in the code
# 3. Commit these changes with the message: "Release: VERSION", create a PR and merge it.
# 4. Add a tag in git to mark the release: "git tag -a VERSION -m 'Adds tag VERSION for pypi' "
#    Push the tag to git:
#      git push --tags origin main
#    It is necessary to work on the original repository, not on a fork.
# 5. Run the following commands in the top-level directory:
#      python setup.py bdist_wheel
#      python setup.py sdist
#    Ensure that you are on the clean and up-to-date main branch (git status --untracked-files=no should not list any
#    files and show the main branch)
# 6. Upload the package to the pypi test server first:
#      twine upload dist/* -r pypitest
# 7. Check that you can install it in a virtualenv by running:
#      pip install -i https://testpypi.python.org/pypi --extra-index-url https://pypi.org/simple peft
# 8. Upload the final version to actual pypi:
#      twine upload dist/* -r pypi
# 9. Add release notes to the tag on https://github.com/huggingface/peft/releases once everything is looking hunky-dory.
#      Check the notes here: https://docs.google.com/document/d/1k-sOIfykuKjWcOIALqjhFKz4amFEp-myeJUJEzNgjoU/edit?usp=sharing
# 10. Update the version in __init__.py, setup.py to the bumped minor version + ".dev0" (e.g. from "0.6.0" to "0.7.0.dev0")