环境:2* A100 40G
模型:qwen2-7B
服务:Flask + gunicorn
单卡推理, gpu_memory_utilization=0.5, 单卡推理耗时5s:
llm = LLM(model='/root/autodl-tmp/model/qwen/Qwen2-7B-Instruct', tokenizer_mode="auto", dtype=torch.bfloat16, gpu_memory_utilization=0.7, enforce_eager=True)
单卡占用内存情况
多卡推理,设置tensor_parallel_size=2,
llm = LLM(model='/root/autodl-tmp/model/qwen/Qwen2-7B-Instruct', tokenizer_mode="auto", dtype=torch.bfloat16, gpu_memory_utilization=0.7, enforce_eager=True, tensor_parallel_size=2)
多卡占用内存
多卡推理结果, 推理耗时11s
为什么多卡推理耗时更长了😵💫