镜像网站及说明:https://hf-mirror.com/
其他教程:如何快速下载huggingface模型——全方法总结
一、huggingface-cli方法下载
1.1安装依赖
pip install -U huggingface_hub
1.2 设置环境变量
linux
export HF_ENDPOINT=https://hf-mirror.com
windows powershell
$env:HF_ENDPOINT = "https://hf-mirror.com"
1.2.1 配置hf-transfer加速(可选
,有时会失败)
开不开 ,下载的命令都是一样的!
pip install -U hf-transfer
export HF_HUB_ENABLE_HF_TRANSFER=1
1.3下载单个项目
模型下载示例1
huggingface-cli download --resume-download gpt2 --local-dir gpt2
项目下载示例
huggingface-cli download --resume-download openai/clip-vit-large-patch14-336 --local-dir ./models/clip-vit-large-patch14-336
1.3.2 数据集
huggingface-cli download --repo-type dataset --resume-download wikitext --local-dir wikitext
示例2
https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain
huggingface-cli download --repo-type dataset --resume-download liuhaotian/LLaVA-Pretrain --local-dir datasets/LLaVA-Pretrain
加速下载后的示意图
1.4 linux批量下载模型 (脚本见附录)
bash cli_download.sh
二、下载需要Access key的(如llama3)
uggingface-cli: 添加–token参数
在官网这里获取 Access Token : https://huggingface.co/settings/tokens
huggingface-cli download --resume-download meta-llama/Meta-Llama-3-8B-Instruct --token your_key --local-dir ./meta-llama/Meta-Llama-3-8B-Instruct
附录
cli_download.sh
export HF_ENDPOINT=https://hf-mirror.com
# Set models and datasets to download
models=(
"nlpconnect/vit-gpt2-image-captioning"
# "lllyasviel/ControlNet"
"lllyasviel/sd-controlnet-canny"
# "lllyasviel/sd-controlnet-depth"
# "lllyasviel/sd-controlnet-hed"
# "lllyasviel/sd-controlnet-mlsd"
# "lllyasviel/sd-controlnet-openpose"
# "lllyasviel/sd-controlnet-scribble"
# "lllyasviel/sd-controlnet-seg"
# "runwayml/stable-diffusion-v1-5"
"damo-vilab/text-to-video-ms-1.7b"
"microsoft/speecht5_asr"
"JorisCos/DCCRNet_Libri1Mix_enhsingle_16k"
"espnet/kan-bayashi_ljspeech_vits"
"facebook/detr-resnet-101"
"microsoft/speecht5_hifigan"
"microsoft/speecht5_vc"
"openai/whisper-base"
"Intel/dpt-large"
"facebook/detr-resnet-50-panoptic"
"facebook/detr-resnet-50"
"google/owlvit-base-patch32"
"impira/layoutlm-document-qa"
"ydshieh/vit-gpt2-coco-en"
"dandelin/vilt-b32-finetuned-vqa"
"lambdalabs/sd-image-variations-diffusers"
"facebook/maskformer-swin-base-coco"
"Intel/dpt-hybrid-midas"
)
datasets=("Matthijs/cmu-arctic-xvectors")
# Download models
for model in "${models[@]}"; do
echo "----- Downloading model ${model} -----"
huggingface-cli download --resume-download "${model}" --local-dir "${model}"
done
# Download datasets
for dataset in "${datasets[@]}"; do
echo "----- Downloading dataset ${dataset} -----"
huggingface-cli download --repo-type dataset --resume-download "datasets/${dataset}" --local-dir "${dataset}"
done