文章讨论了在使用SentencePiece进行文本生成训练时遇到的RuntimeError，指出问题可能在于训练数据格式不正确，建议将数据适当地转换为字典并使用json.dumps处理，确保非ASCII字符被正确编码。
Traceback (most recent call last):
File "/xxxtext_generation_train/preprocess/token_preprocess/train_tokenizer.py", line 170, in <module>
spm.SentencePieceTrainer.train(
File "/usr/local/lib/python3.10/dist-packages/sentencepiece/__init__.py", line 989, in Train
SentencePieceTrainer._Train(arg=arg, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/sentencepiece/__init__.py", line 973, in _Train
model_proto = SentencePieceTrainer._TrainFromMap4(new_kwargs,
File "/usr/local/lib/python3.10/dist-packages/sentencepiece/__init__.py", line 939, in _TrainFromMap4
return _sentencepiece.SentencePieceTrainer__TrainFromMap4(args, iter)
RuntimeError: Internal: src/trainer_interface.cc(428) [!sentences_.empty()]
检查训练数据格式，不要直接print ，要加到dict里面json.dumps(d, ensure_ascii=False)
6709
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
