Coolpi CM5运行ChatGLM-MNN大语言模型

本文基于cool-pi CM5 32G内存版本验证，系统为coolpi官方ubuntu20.04.

最终效果

由于纯CPU跑，实际测试1.5秒/字，内存占用26G左右。

Q: 写一首描绘冬天的诗

A:  漫天雪花飞舞,
寒风呼啸,大地一片沉寂。
冬天的气息弥漫,
银装素裹的世界令人陶醉。

树枝上挂满晶莹剔透的冰霜,
寒风把它们吹得摇曳生姿。
湖面上结满了厚厚的冰,
划着小船的人们欢笑着前行。

冬天是一个美丽而神秘的季节,
让人感受到宁静和祥和。
虽然它带来了一些不便,
比如冷的天气和冰雪的覆盖,
但它也是一年中最美好的时光。<eop>

Q: 

测试步骤

• Download ChatGLM-MNN project

git clone https://github.com/wangzhaode/ChatGLM-MNN.git

• Compile MNN library

git clone https://github.com/alibaba/MNN.git
cd MNN
mkdir build && cd build
cmake .. # CPU only
make -j8
cp -r ../include ../../ChatGLM-MNN/
cp libMNN.so ../../ChatGLM-MNN/libs
cp express/libMNN_Express.so ../../ChatGLM-MNN/libs

• Download Models

cd ChatGLM-MNN/resource/models
./download_models.sh int8
cp int8/* ./

• Build

mkdir build
cd build
cmake ..
make -j8

• Run

coolpi@Ubuntu:~/share/ChatGLM-MNN/build$ ./cli_demo    
The device support i8sdot:1, support fp16:1, support i8mm: 0
Can't Find type=2 backend, use 3 instead
load ../resource/tokenizer/slim_vocab.txt ... Done!
load ../resource/models/glm_block_0.mnn model ... Done!
load ../resource/models/glm_block_1.mnn model ... Done!
load ../resource/models/glm_block_2.mnn model ... Done!
load ../resource/models/glm_block_3.mnn model ... Done!
load ../resource/models/glm_block_4.mnn model ... Done!
load ../resource/models/glm_block_5.mnn model ... Done!
load ../resource/models/glm_block_6.mnn model ... Done!
load ../resource/models/glm_block_7.mnn model ... Done!
load ../resource/models/glm_block_8.mnn model ... Done!
load ../resource/models/glm_block_9.mnn model ... Done!
load ../resource/models/glm_block_10.mnn model ... Done!
load ../resource/models/glm_block_11.mnn model ... Done!
load ../resource/models/glm_block_12.mnn model ... Done!
load ../resource/models/glm_block_13.mnn model ... Done!
load ../resource/models/glm_block_14.mnn model ... Done!
load ../resource/models/glm_block_15.mnn model ... Done!
load ../resource/models/glm_block_16.mnn model ... Done!
load ../resource/models/glm_block_17.mnn model ... Done!
load ../resource/models/glm_block_18.mnn model ... Done!
load ../resource/models/glm_block_19.mnn model ... Done!
load ../resource/models/glm_block_20.mnn model ... Done!
load ../resource/models/glm_block_21.mnn model ... Done!
load ../resource/models/glm_block_22.mnn model ... Done!
load ../resource/models/glm_block_23.mnn model ... Done!
load ../resource/models/glm_block_24.mnn model ... Done!
load ../resource/models/glm_block_25.mnn model ... Done!
load ../resource/models/glm_block_26.mnn model ... Done!
load ../resource/models/glm_block_27.mnn model ... Done!
load ../resource/models/lm.mnn model ... Done!

Q: 写一首描绘冬天的诗

A:  漫天雪花飞舞,
寒风呼啸,大地一片沉寂。
冬天的气息弥漫,
银装素裹的世界令人陶醉。

树枝上挂满晶莹剔透的冰霜,
寒风把它们吹得摇曳生姿。
湖面上结满了厚厚的冰,
划着小船的人们欢笑着前行。

冬天是一个美丽而神秘的季节,
让人感受到宁静和祥和。
虽然它带来了一些不便,
比如冷的天气和冰雪的覆盖,
但它也是一年中最美好的时光。<eop>

Q: