参考
https://llm.mlc.ai/docs/deploy/android.html#android-sdk
环境准备
1. 安装rust (ubuntu)
参考 https://forge.rust-lang.org/infra/other-installation-methods.html#which
export https_proxy=192.168.2.4:21083
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup default stable
export PATH=~/.cargo/bin:$PATH
2. 安装Android Studio
参考 https://developer.android.com/studio
需要在Projects → SDK Manager → SDK Tools 中安装NDK和Cmake
3. 设置环境变量
export ANDROID_NDK=~/Android/Sdk/ndk/27.0.12077973
export TVM_NDK_CC=$ANDROID_NDK/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang
export JAVA_HOME=~/Downloads/android-studio-2024.1.1.12-linux/android-studio/jbr
export PATH=/usr/local/cuda-12/bin:$PATH
export PATH=~/Android/Sdk/cmake/3.22.1/bin:$PATH
export PATH=~/.cargo/bin:$PATH
转换模型
1. 安装mlc-llm
参考https://llm.mlc.ai/docs/install/mlc_llm.html
(提前安装cuda和对应版本的pytorch)
conda create --name mlc-prebuilt python=3.11
conda activate mlc-prebuilt
python -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cu122 mlc-ai-nightly-cu122
conda install -c conda-forge git-lfs
git clone https://github.com/mlc-ai/mlc-llm.git
cd mlc-llm
git submodule update --init --recursive
cd android/MLCChat
export TVM_SOURCE_DIR=~/Documents/mlc-llm/3rdparty/tvm
export MLC_LLM_SOURCE_DIR=~/Documents/mlc-llm
如果报错如下,说明cmake版本太低
CMake Error at ../../3rdparty/tokenizers-cpp/CMakeLists.txt:132 (target_link_libraries): Cannot specify link libraries for target "tokenizers_c" which is not built by this project.
使用Android Studio安装的cmake
export PATH=~/Android/Sdk/cmake/3.22.1/bin:$PATH
2. 克隆模型到本地,运行转换
You can be under the mlc-llm repo, or your own working directory. Note that all platforms can share the same compiled/quantized weights. See Compile Command Specification for specification of convert_weight
.
mlc_llm convert_weight ./dist/models/internlm2-1_8b/ \
--quantization q4f16_1 \
-o dist/internlm2-1_8b-q4f16_1-MLC
如果报错 No such file or directory: ‘nvcc’
export PATH=/usr/local/cuda-12/bin:$PATH
3. 生成配置(可能需要调整)
Use mlc_llm gen_config to generate mlc-chat-config.json and process tokenizers. See Compile Command Specification for specification of gen_config
.
mlc_llm gen_config ./dist/models/internlm2-1_8b/ \
--quantization q4f16_1 --conv-template chatml \
-o dist/internlm2-1_8b-q4f16_1-MLC
4. 编译成二进制文件(用于测试)
mlc_llm compile ./dist/internlm2-1_8b-q4f16_1-MLC/mlc-chat-config.json \
--device cuda -o dist/libs/internlm2-1_8b-q4f16_1-cuda.so
5. 测试编译的模型是否符合预期
(手机端运行的效果和测试效果接近)
from mlc_llm import MLCEngine
# Create engine
engine = MLCEngine(model="./dist/internlm2-1_8b-q4f16_1-MLC", model_lib="./dist/libs/internlm2-1_8b-q4f16_1-MLC-cuda.so")
# Run chat completion in OpenAI API.
print(engine)
for response in engine.chat.completions.create(
messages=[{"role": "user", "content": "你是谁?"}],
stream=True
):
for choice in response.choices:
print(choice.delta.content, end="", flush=True)
print("\n")
engine.terminate()
6. 上传到huggingface
参考 https://huggingface.co/timws/internlm2-1_8b-q4f16_1-MLC
打包运行
1. 修改配置文件mlc-package-config.json
参考如下
{
"device": "android",
"model_list": [
{
"model": "HF://mlc-ai/gemma-2b-it-q4f16_1-MLC",
"model_id": "gemma-2b-q4f16_1-MLC",
"estimated_vram_bytes": 3980990464
}
,
{
"model": "HF://timws/internlm2-1_8b-q4f16_1-MLC",
"model_id": "internlm2-1_8b-q4f16_1-MLC",
"estimated_vram_bytes": 3980990464
}
]
}
2. 运行打包命令
mlc_llm package
3. 编译和运行
使用Android Studio打开./android/MLCChat
点击 “Build → Make Project”进行编译
使用WiFi调试连接手机
点击 “Run → Run ‘app’” 进行调试
在手机上运行时,需要有外部网络访问权限