kaggle运行报错RuntimeError: cutlassF: no kernel found to launch!

项目场景：

项目场景：使用原始Llama3推理，到这里都是能行的

!pip install -q modelscope
import torch
from modelscope import snapshot_download, AutoModel, AutoTokenizer
import os
model_dir = snapshot_download('LLM-Research/Meta-Llama-3-8B-Instruct', cache_dir='/root/autodl-tmp', revision='master')
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
    model_dir, torch_dtype="auto", device_map="auto"
)

---

问题描述

RuntimeError: cutlassF: no kernel found to launch!

messages = [
    {
        'role':'user',
        'content':"""hello"""
    }
]

input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids=input_ids,
    max_new_tokens=8192,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

报错在outputs = model.generate()

---

原因分析：

刚开始怀疑是cuda问题，或者cpu、GPU问题

device = next(model.parameters()).device
print(f"Model is on {device}")

打印出来没有发现问题

---

解决方案：

在网上看到了一个解决方案，完美解决

torch.backends.cuda.enable_mem_efficient_sdp(False)
torch.backends.cuda.enable_flash_sdp(False)

原链接如下：https://stackoverflow.com/questions/77803696/runtimeerror-cutlassf-no-kernel-found-to-launch-when-running-huggingface-tran