首先安装依赖包:
sudo apt-get install autoconf automake libtool libprotobuf9v5 protobuf-compiler libprotobuf-dev
下载sentencepiece : git clone https://github.com/google/sentencepiece
编译与安装sentencepiece:
cd /path/to/sentencepiece
./autogen.sh
./configure
make
make check
sudo make install
sudo ldconfig
训练sentencepiece 模型:
spm_train --input=<<