复制 python cache 文件到预训练模型

最新推荐文章于 2023-08-09 14:29:14 发布
最新推荐文章于 2023-08-09 14:29:14 发布
阅读量1.4k 收藏
点赞数
文章标签： python pytorch 开发语言
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/handavy/article/details/121462773
版权

做测试中使用 hugface的transformers 在线下载了一堆目录的文件,需要整理用于离线使用


 

import os ,sys ,shutil
import re
import json
cache_path='D:\\PYTORCH_TRANSFORMERS_CACHE\\'
bert_pretain='D:\\bert_model\\'
for p,f,s in os.walk(cache_path):
    #print (p)
    for file in s:
        #print(re.split('[.]',file))
        if re.split('[.]',file)[-1]=='json' :
            #print ('.'.join(re.split('[.]',file)[0:-1]))
            target_file='.'.join(re.split('[.]',file)[0:-1])
            j=json.load(open(p+'/'+file))
            print(j['url'])
            model_name=j['url'].split('/resolve/')[0].split('/')[-1]
            model_file_name=j['url'].split('/')[-1]
            print(model_name,model_file_name)
            target_path=bert_pretain+model_name
            #print(target_path)
            if not os.path.exists(target_path):
                os.makedirs(target_path) 
            print(p+target_file)
            print(target_path+'/'+model_file_name)
            if os.path.exists(p+target_file)&~os.path.exists(target_path+'/'+model_file_name):
                shutil.copyfile(p+target_file ,target_path+'/'+model_file_name)  








https://huggingface.co/xlnet-large-cased/resolve/main/pytorch_model.bin
xlnet-large-cased pytorch_model.bin
D:\PYTORCH_TRANSFORMERS_CACHE\0c2b00a768ca7c5b3534b75606a47a7e1125b10ce354b217022de5a12029859c.7fff7afe180c24f31dabdb196f95ca2e26a8aa357c1db6137f4fec6430db9776
D:\bert_model\xlnet-large-cased/pytorch_model.bin
https://huggingface.co/hfl/chinese-electra-180g-base-discriminator/resolve/main/pytorch_model.bin
chinese-electra-180g-base-discriminator pytorch_model.bin
D:\PYTORCH_TRANSFORMERS_CACHE\1b16640da935adb492a0d70765ccca3300c01a7f2ce15991ff648ecd34cd48ed.2b59522b202567cfa0c53424492aa0cfa90f6a5b34205e4f4b21b8b903994ed1
D:\bert_model\chinese-electra-180g-base-discriminator/pytorch_model.bin
https://huggingface.co/wptoux/albert-chinese-large-qa/resolve/main/pytorch_model.bin
albert-chinese-large-qa pytorch_model.bin
D:\PYTORCH_TRANSFORMERS_CACHE\1bf27d86a25788f69cbf644e74289ab3eff3fdeb3bb533bfc96e08aec2250d5c.f9736c09f45fd2d06cf369c230c49d986759f331cae064f5b1d203a1639a09f6
D:\bert_model\albert-chinese-large-qa/pytorch_model.bin
https://huggingface.co/xlnet-large-cased/resolve/main/config.json
xlnet-large-cased config.json
D:\PYTORCH_TRANSFORMERS_CACHE\1f0d5fc4143aa8fe332810bac98d442fed5483549adcd9656e5709cd470003a0.a0945cddd1ef8f9d9c40c35c36bad4908625533057baeeafc6d26a9550f18c60
D:\bert_model\xlnet-large-cased/config.json
https://huggingface.co/wptoux/albert-chinese-large-qa/resolve/main/special_tokens_map.json
albert-chinese-large-qa special_tokens_map.json
D:\PYTORCH_TRANSFORMERS_CACHE\1f536158f09966b7ddca1f2c06264b78535edb420d115380141a23361d136d78.dd8bd9bfd3664b530ea4e645105f557769387b3da9f79bdb55ed556bdd80611d
D:\bert_model\albert-chinese-large-qa/special_tokens_map.json
https://huggingface.co/hfl/chinese-electra-180g-base-discriminator/resolve/main/added_tokens.json
chinese-electra-180g-base-discriminator added_tokens.json
D:\PYTORCH_TRANSFORMERS_CACHE\215afa37f85d31a1dc760ee7ea2b40414d6f3b518a20ce319a66bdd9a79ca510.5cc6e825eb228a7a5cfd27cb4d7151e97a79fb962b31aaf1813aa102e746584b
D:\bert_model\chinese-electra-180g-base-discriminator/added_tokens.json
https://huggingface.co/hfl/chinese-electra-180g-base-discriminator/resolve/main/vocab.txt
chinese-electra-180g-base-discriminator vocab.txt
D:\PYTORCH_TRANSFORMERS_CACHE\23fd079e75a41fee404f2fe8a5b7899af4afbae037b66ad3b79f23023f1e50be.accd894ff58c6ff7bd4f3072890776c14f4ea34fcc08e79cd88c2d157756dceb
D:\bert_model\chinese-electra-180g-base-discriminator/vocab.txt
https://huggingface.co/kssteven/ibert-roberta-base/resolve/main/pytorch_model.bin
ibert-roberta-base pytorch_model.bin
D:\PYTORCH_TRANSFORMERS_CACHE\27deb62eabe44acce2863c490a8dcbeb98053eeb8a0275c10326865020aaac43.c555a2e298657aa872e144ac44cbd115f4825d74d48414d00781f2fd00cbb3cd
D:\bert_model\ibert-roberta-base/pytorch_model.bin
https://huggingface.co/hfl/chinese-electra-180g-base-discriminator/resolve/main/tokenizer.json
chinese-electra-180g-base-discriminator tokenizer.json
D:\PYTORCH_TRANSFORMERS_CACHE\2c3d89a81a0b01080027739ef180adbee95025e6003a8992fa4980ef0a9463ec.660ed5c7513bf13d4607410502a84e0de517eb889ff8c401068a1688868e1ccb
D:\bert_model\chinese-electra-180g-base-discriminator/tokenizer.json
https://huggingface.co/kssteven/ibert-roberta-base/resolve/main/tokenizer.json
ibert-roberta-base tokenizer.json
D:\PYTORCH_TRANSFORMERS_CACHE\34f3d5049e70c4f56a6bd42bd6a9e4f2599164010440a37895e99aafb084c114.fc9576039592f026ad76a1c231b89aee8668488c671dfbe6616bab2ed298d730
D:\bert_model\ibert-roberta-base/tokenizer.json
https://huggingface.co/xlnet-large-cased/resolve/main/spiece.model
xlnet-large-cased spiece.model
D:\PYTORCH_TRANSFORMERS_CACHE\3af982b422f8bb8c510fdd1112afe6f5ec3f3219ef859edcf4c3826bec14832e.d93497120e3a865e2970f26abdf7bf375896f97fde8b874b70909592a6c785c9
D:\bert_model\xlnet-large-cased/spiece.model
https://huggingface.co/kssteven/ibert-roberta-base/resolve/main/tokenizer_config.json
ibert-roberta-base tokenizer_config.json
D:\PYTORCH_TRANSFORMERS_CACHE\45e95c68c39ff43eaa320e2634280dc9be9fbeac498575241b162472b280a60f.e7fcf26aa8cb28b14292f4ba90abfc66937ea92782061f8552cbd61edc0f7c0a
D:\bert_model\ibert-roberta-base/tokenizer_config.json
https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/config.json
distilbert-base-uncased-finetuned-sst-2-english config.json
D:\PYTORCH_TRANSFORMERS_CACHE\4e60bb8efad3d4b7dc9969bf204947c185166a0a3cf37ddb6f481a876a3777b5.9f8326d0b7697c7fd57366cdde57032f46bc10e37ae81cb7eb564d66d23ec96b
D:\bert_model\distilbert-base-uncased-finetuned-sst-2-english/config.json
https://huggingface.co/roberta-base/resolve/main/pytorch_model.bin
roberta-base pytorch_model.bin
D:\PYTORCH_TRANSFORMERS_CACHE\51ba668f7ff34e7cdfa9561e8361747738113878850a7d717dbc69de8683aaad.c7efaa30a0d80b2958b876969faa180e485944a849deee4ad482332de65365a7
D:\bert_model\roberta-base/pytorch_model.bin
https://huggingface.co/kssteven/ibert-roberta-base/resolve/main/merges.txt
ibert-roberta-base merges.txt
D:\PYTORCH_TRANSFORMERS_CACHE\5422ece434498216e797c6ef1ecef875e497cb88774ffb57341843b89567fa70.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b
D:\bert_model\ibert-roberta-base/merges.txt
https://huggingface.co/wptoux/albert-chinese-large-qa/resolve/main/config.json
albert-chinese-large-qa config.json
D:\PYTORCH_TRANSFORMERS_CACHE\61055dc1dfe98ea7e5028eb70dc8819823b10e8a13486f9176c5f28cb6c1336a.79acb9e30f2e23c7e73eb49d29e01e9fe406e2032f21e044939223d70207639e
D:\bert_model\albert-chinese-large-qa/config.json
https://huggingface.co/kssteven/ibert-roberta-base/resolve/main/vocab.json
ibert-roberta-base vocab.json
D:\PYTORCH_TRANSFORMERS_CACHE\673a127a1efd88da2f306da00117064b72a5cc86bcca7be220d81c9c74369858.647b4548b6d9ea817e82e7a9231a320231a1c9ea24053cc9e758f3fe68216f05
D:\bert_model\ibert-roberta-base/vocab.json
https://huggingface.co/xlnet-large-cased/resolve/main/tokenizer.json
xlnet-large-cased tokenizer.json
D:\PYTORCH_TRANSFORMERS_CACHE\6a4afd4829edeea0c7fe7735eccea233e66e79729e574966cfd9ec47f81d269a.2a683f915238b4f560dab0c724066cf0a7de9a851e96b0fb3a1e7f0881552f53
D:\bert_model\xlnet-large-cased/tokenizer.json
https://huggingface.co/roberta-base/resolve/main/config.json
roberta-base config.json
D:\PYTORCH_TRANSFORMERS_CACHE\733bade19e5f0ce98e6531021dd5180994bb2f7b8bd7e80c7968805834ba351e.35205c6cfc956461d8515139f0f8dd5d207a2f336c0c3a83b4bc8dca3518e37b
D:\bert_model\roberta-base/config.json
https://huggingface.co/hfl/chinese-electra-180g-base-discriminator/resolve/main/special_tokens_map.json
chinese-electra-180g-base-discriminator special_tokens_map.json
D:\PYTORCH_TRANSFORMERS_CACHE\76aef35396bf3e77a19ce4972226da9d23226e3c3b6319cf3d232ec1ce3e9031.dd8bd9bfd3664b530ea4e645105f557769387b3da9f79bdb55ed556bdd80611d
D:\bert_model\chinese-electra-180g-base-discriminator/special_tokens_map.json
https://huggingface.co/distilbert-base-cased-distilled-squad/resolve/main/config.json
distilbert-base-cased-distilled-squad config.json
D:\PYTORCH_TRANSFORMERS_CACHE\81e8dfe090123eff18dd06533ead3ae407b82e30834c50c7c82c2305ce3ace12.ca0305b1f128274fa0c6e4859d1c1477d0e34a20be25da95eea888b30ece9cf3
D:\bert_model\distilbert-base-cased-distilled-squad/config.json
https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/vocab.txt
distilbert-base-uncased-finetuned-sst-2-english vocab.txt
D:\PYTORCH_TRANSFORMERS_CACHE\83261b0c74c462e53d6367de0646b1fca07d0f15f1be045156b9cf8c71279cc9.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99
D:\bert_model\distilbert-base-uncased-finetuned-sst-2-english/vocab.txt
https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/pytorch_model.bin
distilbert-base-uncased-finetuned-sst-2-english pytorch_model.bin
D:\PYTORCH_TRANSFORMERS_CACHE\8d04c767d9d4c14d929ce7ad8e067b80c74dbdb212ef4c3fb743db4ee109fae0.9d268a35da669ead745c44d369dc9948b408da5010c6bac414414a7e33d5748c
D:\bert_model\distilbert-base-uncased-finetuned-sst-2-english/pytorch_model.bin
https://huggingface.co/wptoux/albert-chinese-large-qa/resolve/main/vocab.txt
albert-chinese-large-qa vocab.txt
D:\PYTORCH_TRANSFORMERS_CACHE\b4e2bf6a6135d917a3ad4ddfe65184836aa367fa73d96161ba02f8f0a3cd07d4.accd894ff58c6ff7bd4f3072890776c14f4ea34fcc08e79cd88c2d157756dceb
D:\bert_model\albert-chinese-large-qa/vocab.txt
https://huggingface.co/kssteven/ibert-roberta-base/resolve/main/special_tokens_map.json
ibert-roberta-base special_tokens_map.json
D:\PYTORCH_TRANSFORMERS_CACHE\c2fbe5b3fb8f721fc3f4d0e8e2b8d2ed77db69bef5aaf6779bbee917930b1c95.cb2244924ab24d706b02fd7fcedaea4531566537687a539ebb94db511fd122a0
D:\bert_model\ibert-roberta-base/special_tokens_map.json
https://huggingface.co/roberta-base/resolve/main/merges.txt
roberta-base merges.txt
D:\PYTORCH_TRANSFORMERS_CACHE\cafdecc90fcab17011e12ac813dd574b4b3fea39da6dd817813efa010262ff3f.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b
D:\bert_model\roberta-base/merges.txt
https://huggingface.co/kssteven/ibert-roberta-base/resolve/main/config.json
ibert-roberta-base config.json
D:\PYTORCH_TRANSFORMERS_CACHE\cfb510f67e8b7caa315edb63cf273dcabea566cc7c79256c9279b9aabfabc1e2.6e328a8b48a360bcdc4fa4628970901425656415d87641bc286c517e3f274c05
D:\bert_model\ibert-roberta-base/config.json
https://huggingface.co/roberta-base/resolve/main/vocab.json
roberta-base vocab.json
D:\PYTORCH_TRANSFORMERS_CACHE\d3ccdbfeb9aaa747ef20432d4976c32ee3fa69663b379deb253ccfce2bb1fdc5.d67d6b367eb24ab43b08ad55e014cf254076934f71d832bbab9ad35644a375ab
D:\bert_model\roberta-base/vocab.json
https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/tokenizer_config.json
distilbert-base-uncased-finetuned-sst-2-english tokenizer_config.json
D:\PYTORCH_TRANSFORMERS_CACHE\d44ec0488a5f13d92b3934cb68cc5849bd74ce63ede2eea2bf3c675e1e57297c.627f9558061e7bc67ed0f516b2f7efc1351772cc8553101f08748d44aada8b11
D:\bert_model\distilbert-base-uncased-finetuned-sst-2-english/tokenizer_config.json
https://huggingface.co/roberta-base/resolve/main/tokenizer.json
roberta-base tokenizer.json
D:\PYTORCH_TRANSFORMERS_CACHE\d53fc0fa09b8342651efd4073d75e19617b3e51287c2a535becda5808a8db287.fc9576039592f026ad76a1c231b89aee8668488c671dfbe6616bab2ed298d730
D:\bert_model\roberta-base/tokenizer.json
https://huggingface.co/hfl/chinese-electra-180g-base-discriminator/resolve/main/config.json
chinese-electra-180g-base-discriminator config.json
D:\PYTORCH_TRANSFORMERS_CACHE\d820a4640dd0941f13efa75aaf0e618feaa452ae3dc1273aab0ccaf2518dad1a.a2041ee12db6173e5336e826e864a87202039f99cee50c6598f43bec207a9e10
D:\bert_model\chinese-electra-180g-base-discriminator/config.json
https://huggingface.co/hfl/chinese-electra-180g-base-discriminator/resolve/main/tokenizer_config.json
chinese-electra-180g-base-discriminator tokenizer_config.json
D:\PYTORCH_TRANSFORMERS_CACHE\e4a342dc9852a75edad532b36968addf2b6d0da241b6938a99de2a0a9e37667f.d23f50bbddc3fb34db5a76d47fa9bdd5d75bf4201ad2d49abbcca25629b3e562
D:\bert_model\chinese-electra-180g-base-discriminator/tokenizer_config.json
https://huggingface.co/wptoux/albert-chinese-large-qa/resolve/main/tokenizer_config.json
albert-chinese-large-qa tokenizer_config.json
D:\PYTORCH_TRANSFORMERS_CACHE\f09a2ad95ee81ac29d743d3147c78b8c2946c31331e7feb2735e09593e903aeb.56af104a95626511966a55b49534621585cf3139a6b17691620ba971664ac74b
D:\bert_model\albert-chinese-large-qa/tokenizer_config.json
handavy
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
pytorch 将数据集加载到内存后再训练
码到功成的博客
07-01 4388
当我们数据比较大时，加载数据，然后再将数据丢到模型中训练时，有时候由于IO的原因，绝大部分时间放在了加载数据上，这时为了提高模型的训练速度，，详细间代码下面是一个图片分类的数据集 原理很简单，在第一次读取数据的时候就一边读取，一边将数据加载到内存中，所以如果是第一次迭代的时候，还是很慢的，这个时候迭代的时候并没有从内存中读取数据，所以在使用的时候，我会先对数据加载一边，但这时候什么都不做，不将数据丢到网络中训练，只为将数据加载到内存中，具体看下面代码用法和常规的数据使用方法有点小区别 上面是正常操作，目的
pkuseg - 最全的预训练模型
06-24
art.zip ctb8.zip default v2.zip entertainment.zip mixed.zip msra.zip news.zip pkuseg-python-master.zip postag.zip science.zip THULAC tourism.zip web.zip weibo.zip
Python 实现cache功能
Yohann丶blog
04-12 3041
WechatIMG911.jpeg 创建 cache.py，代码如下 # -*- coding: utf-8 -*- # !python3 import hashlib import os import pickle cache_root_dir = 'cache' if not os.path.exists(cache_root_dir): os.makedirs(ca...
Cache在Python中的实现-AOP内置实现
flyfoxs的专栏
08-07 966
Python自带的AOP Cache JAVA中可以通过AOP很容易的实现方法的cache, 其实Python比JAVA做的更进一步,直接在核心库中就实现了. 示例 from functools import lru_cache import time import numpy as np #maxsiez设置了这个方法的case size只有2个,超过就会删除最近最少使用 @lr...
HuggingFace学习笔记-零碎记录
gimlet_的博客
08-04 346
例如，如果您使用`from transformers import GPT2LMHeadModel`来加载`distilgpt2`模型，则可能需要将代码修改为`from transformers import AutoModelForCausalLM`或者`from transformers import GPT2LMHeadModel`。您可以使用`transformers`库的`pretrained_models`模块来下载所需的模型权重，具体可以参考相关文档或示例代码。不过有个奇怪的地方。
python-pytorch基础之下载huggingface模型并本地使用，以models--distilbert-base-uncased-finetuned-sst-2-english为例
随笔
08-08 1639
python-pytorch基础之下载huggingface模型并本地使用，以models--distilbert-base-uncased-finetuned-sst-2-english为例
基于python的中文预训练RoBERTa模型型
03-21
**基于Python的中文预训练RoBERTa模型** 在自然语言处理（NLP）领域，预训练模型已经成为理解和生成人类语言的关键技术。RoBERTa（Robustly Optimized BERT Pretraining Approach）是BERT（Bidirectional Encoder ...
基于python的中文预训练生成模型代码实现
03-25
总结，基于Python的中文预训练生成模型代码实现涉及到模型选择、数据处理、模型训练和微调等多个环节。Python的灵活性和丰富的库资源使得这一过程相对简化，但也需要对深度学习和自然语言处理有深入理解。通过实践，...
基于python的高质量中文预训练模型
03-20
其中，`TensorFlow`、`PyTorch`、`Keras`等深度学习框架为构建和训练预训练模型提供了便利。Python的生态使得集成和优化预训练模型变得简单，便于快速实现和迭代。 **大模型与小模型** 大模型，如BERT、GPT系列、...
基于bert的幽默识别预训练模型
07-04
幽默是一种特殊的语言表达方式，在日常生活中扮演着化解尴尬、活跃气氛、促进交流的重要角色。而幽默计算是近年来自然语言处理领域的新兴热点之一，其主要...本资源是基于基于bert的幽默识别模型，请结合我的博客使用！
运行rviz时出现[ERROR]: Could not load model、Cannot locate resource package错误
A080807的博客
12-28 5320
问题 运行solidworks导出的urdf功能包下的display.launch，单独打开一个rviz去接收模型和tf数据时，出现如下错误： [ERROR] [1640699688.189865241]: Could not load model 'package://agv_model/meshes/base_link.STL' for link 'base_link': OGRE EXCEPTION(6:FileNotFoundException): Cannot locate resource p
huggingface NLP工具包教程2：使用Transformers
weixin_44966641的博客
11-01 1237
huggingface NLP工具包教程2：使用Transformers
python-pytorch基础之加载bert模型获取字向量
最新发布
随笔
08-09 274
python-pytorch基础之加载模型获取字向量
深度学习笔记--本地部署Mini-GPT4
牵一只蜗牛去散步
05-03 6767
升级 bitsandbytes 库，这里博主选用 0.38.1 版本的 bitsandbytes 解决了问题；--delta-path 表示第 3 步中下载的 vicuna-7b-delta-v1.1 权重的存放地址；--bash-model-path 表示第 3 步中下载的 llama-7b-hf 权重的存放地址；--target-model-path 表示生成 Vicuna 权重的存放地址；4--生成 Vicuna 权重。6--可能出现的问题。
Hugging Face 的 Transformers 库快速入门 （二）模型与分词器
liu_chengwei的博客
10-20 4043
在上一篇文章《开箱即用的 pipelines》中，我们通过 Transformers 库提供的 pipeline 函数展示了 Transformers 库能够完成哪些 NLP 任务，以及这些 pipelines 背后的工作原理。本文将深入介绍 Transformers 库中的两个重要组件：模型（Models类）和分词器（Tokenizers类）。
Hugging Face 模型下载方法一览。
热门推荐
YI_SHU_JIA的博客
10-24 2万+
hugging face 你不可或缺的工具
mt5模型从spiece.model中找寻出vocab
小姑仔的博客
11-30 1763
mt5模型文件地址 mt5模型由于vocab词汇表中的词汇过多，采用从spiece.model之中读取的方式，读取过程如下 import sentencepiece as spm spm_path = '/home/xiaoguzai/模型/mt5/spiece.model' # 加载训练好的模型，切分文本 sp = spm.SentencePieceProcessor(model_file=spm_path) # 编码 text -> id result = sp.encode(['This i
手把手快速搭建智能语音客服——保险问答实践
m0_63642362的博客
09-27 1689
手把手快速搭建简易的智能语音客服，包含语音交互技术，问答查询技术，零基础快速入门语音识别，语音合成，问答检索
工程师提效神器，有了这个插件，可以自动写代码了
m0_63642362的博客
10-29 1663
领先的代码生成模型、基于PaddleNLP的模型加速能力、以及一整套插件开发流程，来了！
用python创建一个预训练模型
04-18
您可以使用Python中的各种深度学习框架（如PyTorch和TensorFlow）来创建和训练预训练模型。您可以首先选择和下载您想要使用的预训练模型，或者使用数据集来训练您自己的预训练模型。然后，您可以编写Python代码来...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值