【hugggingface+下载】批量加速下载HuggingFace上的模型

曾小蛙

已于 2024-04-29 14:52:52 修改

阅读量2k

点赞数 11

分类专栏： AIGC 生成式人工智能文章标签： huggingface llama3 datasets models

于 2024-03-06 15:25:41 首次发布

本文链接：https://blog.csdn.net/imwaters/article/details/136501247

版权

AIGC 生成式人工智能专栏收录该内容

28 篇文章 7 订阅

订阅专栏

镜像网站及说明：https://hf-mirror.com/
其他教程：如何快速下载huggingface模型——全方法总结

一、huggingface-cli方法下载

1.1安装依赖

pip install -U huggingface_hub

1.2 设置环境变量

linux

export HF_ENDPOINT=https://hf-mirror.com

windows powershell

$env:HF_ENDPOINT = "https://hf-mirror.com"

1.2.1 配置hf-transfer加速（`可选`，有时会失败）

开不开，下载的命令都是一样的！

pip install -U hf-transfer
export HF_HUB_ENABLE_HF_TRANSFER=1

在这里插入图片描述

1.3下载单个项目

模型下载示例1

huggingface-cli download --resume-download gpt2 --local-dir gpt2

项目下载示例

huggingface-cli download --resume-download openai/clip-vit-large-patch14-336  --local-dir ./models/clip-vit-large-patch14-336

1.3.2 数据集

huggingface-cli  download --repo-type dataset --resume-download wikitext --local-dir wikitext

示例2
https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain

huggingface-cli  download --repo-type dataset --resume-download liuhaotian/LLaVA-Pretrain --local-dir datasets/LLaVA-Pretrain

加速下载后的示意图

1.4 linux批量下载模型 (脚本见附录)

bash cli_download.sh

在这里插入图片描述

二、下载需要Access key的（如llama3）

uggingface-cli：添加–token参数
在官网这里获取 Access Token : https://huggingface.co/settings/tokens


huggingface-cli download --resume-download meta-llama/Meta-Llama-3-8B-Instruct   --token  your_key  --local-dir  ./meta-llama/Meta-Llama-3-8B-Instruct

附录

cli_download.sh

export HF_ENDPOINT=https://hf-mirror.com

# Set models and datasets to download
models=(
    "nlpconnect/vit-gpt2-image-captioning"
    # "lllyasviel/ControlNet"
    "lllyasviel/sd-controlnet-canny"
    # "lllyasviel/sd-controlnet-depth"
    # "lllyasviel/sd-controlnet-hed"
    # "lllyasviel/sd-controlnet-mlsd"
    # "lllyasviel/sd-controlnet-openpose"
    # "lllyasviel/sd-controlnet-scribble"
    # "lllyasviel/sd-controlnet-seg"
    # "runwayml/stable-diffusion-v1-5"
    "damo-vilab/text-to-video-ms-1.7b"
    "microsoft/speecht5_asr"
    "JorisCos/DCCRNet_Libri1Mix_enhsingle_16k"
    "espnet/kan-bayashi_ljspeech_vits"
    "facebook/detr-resnet-101"
    "microsoft/speecht5_hifigan"
    "microsoft/speecht5_vc"
    "openai/whisper-base"
    "Intel/dpt-large"
    "facebook/detr-resnet-50-panoptic"
    "facebook/detr-resnet-50"
    "google/owlvit-base-patch32"
    "impira/layoutlm-document-qa"
    "ydshieh/vit-gpt2-coco-en"
    "dandelin/vilt-b32-finetuned-vqa"
    "lambdalabs/sd-image-variations-diffusers"
    "facebook/maskformer-swin-base-coco"
    "Intel/dpt-hybrid-midas"
)
datasets=("Matthijs/cmu-arctic-xvectors")


# Download models
for model in "${models[@]}"; do
    echo "----- Downloading model ${model} -----"
    huggingface-cli download --resume-download "${model}" --local-dir "${model}"
done

# Download datasets
for dataset in "${datasets[@]}"; do
    echo "----- Downloading dataset ${dataset} -----"
    huggingface-cli download --repo-type dataset --resume-download "datasets/${dataset}" --local-dir "${dataset}"
done