预训练模型微调时遇到，CUDA error: device-side assert triggered

第一次用torch跑预训练模型（GPT2LMHeadModel）报错了：
RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

我的代码：

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer
from torch.utils.data import DataLoader, TensorDataset
import torch.optim as optim
import numpy as np

# Example input and target waveforms
input_waveforms = np.random.randn(1000, 100)  # Replace with your input waveforms
target_waveforms = np.random.randn(1000, 100)  # Replace with your target waveforms

# Convert input and target waveforms to tensors
input_waveforms_tensor = torch.tensor(input_waveforms, dtype=torch.float32)
target_waveforms_tensor = torch.tensor(target_waveforms, dtype=torch.float32)

# Define the batch size
batch_size = 8

# Create a TensorDataset
dataset = TensorDataset(input_waveforms_tensor, target_waveforms_tensor)

# Create a DataLoader for batch processing
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)

# Load pre-trained GPT model and tokenizer
model_name = 'gpt2-medium'  # Choose the desired GPT model
model = GPT2LMHeadModel.from_pretrained(model_name)
tokenizer = GPT2Tokenizer.from_pretrained(model_name)

# Set the device (GPU if available)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)

# Define loss function and optimizer
criterion = torch.nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=1e-4)

# Fine-tuning loop
num_epochs = 10

for epoch in range(num_epochs):
    running_loss = 0.0

    for inputs, targets in dataloader:
        inputs = inputs.to(device)
        targets = targets.to(device)

        # Forward pass
        outputs = model(inputs)[0]

        # Compute loss
        loss = criterion(outputs, targets)

        # Backward pass and optimization
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

    epoch_loss = running_loss / len(dataloader)
    print(f'Epoch [{epoch + 1}/{num_epochs}], Loss: {epoch_loss:.4f}')

# Evaluation
test_input_waveforms = np.random.randn(100, 100)  # Replace with your test input waveforms
test_input_waveforms_tensor = torch.tensor(test_input_waveforms, dtype=torch.float32).to(device)

# Convert test input waveforms tensor to the appropriate scalar type
test_input_waveforms_tensor = test_input_waveforms_tensor.to(torch.long)

generated_waveforms = model.generate(test_input_waveforms_tensor, max_length=100)

# Convert generated waveforms from tensors to numpy arrays
generated_waveforms = generated_waveforms.cpu().numpy()

# Print the generated waveforms
print(generated_waveforms)

网上的调整方法还挺多的，我一个一个试一下

1.加入下面这段代码，会打印出比较完整的报错信息。

import os
os.environ["CUDA_LAUNCH_BLOCKING"] = "1"

加了但是没有打印出来

2.把device 改成了device='cpu'，放到cpu上运行，真的报错原因是：

IndexError:index out of range in self

再接着翻发现是输入数据超出了tokenizer的编码范围，回去看一下编码的结果。要往字典里加字之类的。