llama factory 大数据量下训练失败

路人与大师

已于 2024-06-13 15:57:08 修改

阅读量1.9k

点赞数 1

文章标签： llama python 开发语言

于 2024-06-11 15:27:47 首次发布

本文链接：https://blog.csdn.net/weixin_41046245/article/details/139600468

版权

06/11/2024 07:09:50 - WARNING - llmtuner.data.utils - Checksum failed: missing SHA-1 hash value in dataset_info.json.
06/11/2024 07:09:50 - WARNING - llmtuner.data.utils - Checksum failed: missing SHA-1 hash value in dataset_info.json.
Traceback (most recent call last):
  File "/home/ca2/anaconda3/envs/llama/lib/python3.10/site-packages/datasets/builder.py", line 1973, in _prepare_split_single
    for _, table in generator:
  File "/home/ca2/anaconda3/envs/llama/lib/python3.10/site-packages/datasets/packaged_modules/json/json.py", line 122, in _generate_tables
    io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
  File "pyarrow/_json.pyx", line 52, in pyarrow._json.ReadOptions.__init__
  File "pyarrow/_json.pyx", line 77, in pyarrow._json.ReadOptions.block_size.__set__
Over