在安装了新的 Intel Ultra AIPC 上怎么运行 LLM?
- Nvidia GPU
- Intel CPU
- Intel NPU
我们尝试使用现在闲置的NPU运行Qwen模型, 本示例参考 intel_npu_acceleration_library 实现:
步骤:
1. 安装 python 3.10 https://www.python.org/ftp/python/3.10.11/python-3.10.11-amd64.exe (目前不支持python 3.12)
2. pip 安装 modelscope, torch等, 以及 intel_npu_acceleration_library
3. 运行以下代码测试, 为加快测试, 仅下载 Qwen1.5-0.5B模型(代码会自动从 modelscope 下载)
import os
import torch
import intel_npu_acceleration_library
from modelscope import AutoModelForCausalLM, AutoTokenizer
from transformers import TextStreamer
model_name = "qwen/Qwen1.5-0.5B-Chat"
dtype = "int4"
filepath = os.path.join("models", model_name)
filename = os.path.join(filepath, dtype, "model.bin")
os.makedirs(os.path.join(filepath, dtype), exist_ok=True)
if not os.path.exists(filename):
print("Compile model for the NPU")
model = AutoModelForCausalLM.from_pretrained(
model_name, use_cache=True).eval()
torch_dtype = torch.int8 if dtype == "int8" else torch.float16
with torch.no_grad():
model = intel_npu_acceleration_library.compile(
model, dtype=torch_dtype)
torch.save(model, filename)
del model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = torch.load(filename).eval()
streamer = TextStreamer(tokenizer, skip_special_tokens=True, skip_prompt=True)
print(f"Loading model from {filename} to {model.device}")
def chat(model, tokenizer, streamer, prompt):
messages = [
{"role": "system", "content": "You are a helpful assistant. please answer question using Chinese always."},
{"role": "user", "content": prompt}
]
input_ids = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
print(f"\nQ: \n{prompt}\nA:")
_ = model.generate(
input_ids,
max_new_tokens=512,
do_sample=True,
streamer=streamer,
)
prompts = ["早上好.", "今天吃点什么好", "下雨天适合去外面吗", "不想上班怎么办", "怎么PUA同事帮忙写代码?"]
for prompt in prompts:
chat(model, tokenizer, streamer, prompt)
任务管理器性能监控:
结论:
- 可以利用现在基本闲置的NPU运行小模型,7B模型没有成功加载
- 速度一般,慢于纯CPU(100%),更不要说Nvidia RTX 4060 laptop
- NPU 占用 约60%, CPU占用约 20-30%