Linux环境下部署阿里通义千问开源AI大模型-基于厚德云

 算力环境配置参考：

Linux环境下部署百度百川AI大模型-基于厚德云-CSDN博客

  服务器环境

显存：24 GB
CPU：16 核   
内存：100 GB
总存储: 160 GB

软件环境

git
python 3.10.13
pytorch 2.4.0

 算力平台

厚德云

步骤一：下载Qwen模型

mkdir /workspace/
cd /workspace/
git  clone https://github.moeyy.xyz/https://github.com/QwenLM/Qwen

步骤二：下载依赖

cd Qwen
pip install -r requirements.txt
pip install -r requirements_web_demo.txt
pip install bitsandbytes==0.41.1
pip install accelerate==0.25.0
pip install streamlit
sudo apt update
sudo apt install git-lfs

步骤三：下载模型库

mkdir Qwen
cd Qwen
git lfs clone https://www.modelscope.cn/ccyh123/Qwen-7B-Chat.git

步骤四：启动并应用大模型

cd /workspace/Qwen/
python cli_demo.py

步骤五：网页版启动

cd /workspace/Qwen/
streamlit run web_demo.py

步骤六：使用Qwen大模型

 

Streamlithttp://211.93.18.69:64123/

或

Streamlithttp://zhy1.dc.houdeyun.cn:64123/ 

FAQ

1、OutOfMemoryError: CUDA out of memory. Tried to allocate 64.00 MiB. GPU 0 has a total capacity of 23.65 GiB of which 15.81 MiB is free. Including non-PyTorch memory, this process has 23.63 GiB memory in use. Of the allocated memory 23.25 GiB is allocated by PyTorch, and 328.50 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

解决方案：

显存24G内存不足，升级GPU 

2、OSError: Cannot find empty port in range: 8000-8000. You can specify a different port by setting the GRADIO_SERVER_PORT environment variable or passing the `server_port` parameter to `launch()`.
IMPORTANT: You are using gradio version 3.41.2, however version 4.29.0 is available, please upgrade.
解决方案

修改/workspace/Qwen/web_demo.py文件的第31行，端口号改成其它，比如8010