使用transformers模型时遇到文件损坏问题

myccver

已于 2024-09-27 14:11:24 修改

阅读量977

点赞数 2

文章标签：深度学习

于 2024-07-27 14:23:33 首次发布

本文链接：https://blog.csdn.net/qq_45270993/article/details/140734961

版权

使用transformers模型时遇到以下报错

Loading checkpoint shards:   0%|                                                                                                                               | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/site-packages/transformers/modeling_utils.py", line 460, in load_state_dict
    return torch.load(checkpoint_file, map_location="cpu")
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/site-packages/torch/serialization.py", line 789, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/site-packages/torch/serialization.py", line 1131, in _load
    result = unpickler.load()
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/site-packages/torch/serialization.py", line 1101, in persistent_load
    load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/site-packages/torch/serialization.py", line 1079, in load_tensor
    storage = zip_file.get_storage_from_record(name, numel, torch.UntypedStorage).storage().untyped()
RuntimeError: PytorchStreamReader failed reading file data/43: invalid header or archive is corrupted

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/site-packages/transformers/modeling_utils.py", line 464, in load_state_dict
    if f.read(7) == "version":
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 128: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yunchuan/ceshi/blip2_test.py", line 9, in <module>
    model = Blip2ForConditionalGeneration.from_pretrained(
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3246, in _load_pretrained_model
    state_dict = load_state_dict(shard_file)
  File "/home/yunchuan/miniconda3/envs/lavad/lib/python3.10/site-packages/transformers/modeling_utils.py", line 476, in load_state_dict
    raise OSError(
OSError: Unable to load weights from pytorch checkpoint file for '/home/yunchuan/.cache/huggingface/transformers/Salesforce/blip2-flan-t5-xl-coco/pytorch_model-00001-of-00002.bin' at '/home/yunchuan/.cache/huggingface/transformers/Salesforce/blip2-flan-t5-xl-coco/pytorch_model-00001-of-00002.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

检查了下是模型下载时损坏的问题
可用以下代码检查模型文件是否损坏

import torch
state_dict = torch.load('/home/yunchuan/.cache/huggingface/transformers/Salesforce/blip2-flan-t5-xl-coco/pytorch_model-00002-of-00002.bin', map_location=torch.device('cpu'))

如果报错证明文件损坏

除此之外，还可以用以下代码加载safetensors文件验证是否损坏:

import torch
from safetensors import safe_open

with safe_open('/home/share3/mayunchuan/meta-llama/Llama-3.2-11B-Vision-Instruct/model-00005-of-00005.safetensors', framework="pt", device='cuda:0') as f:
    for k in f.keys():
        print(k)