【Llama3源码】llama3-implemented-from-scratch源码运行踩坑Incorrect padding

Error: Incorrect padding

问题描述

学习naklecha/llama3-from-scratch代码时，首先要下载Meta-Llama-3-8B模型文件

in this file, i implemented llama3 from scratch, one tensor and matrix multiplication at a time. also, im going to load tensors directly from the model file that meta provided for llama3, you need to download the weights before running this file. here is the offical link to download the weights: https://llama.meta.com/llama-downloads/

结合上述说明以及Llama 3 使用方法以及模型下载教程链接方式下载Meta-Llama-3-8B模型文件后，运行下述代码：

from pathlib import Path
import tiktoken
from tiktoken.load import load_tiktoken_bpe
import torch
import json
import matplotlib.pyplot as plt

tokenizer_path = "Meta-Llama-3-8B/tokenizer.model"
special_tokens = [
            "<|begin_of_text|>",
            "<|end_of_text|>",
            "<|reserved_special_token_0|>",
            "<|reserved_special_token_1|>",
            "<|reserved_special_token_2|>",
            "<|reserved_special_token_3|>",
            "<|start_header_id|>",
            "<|end_header_id|>",
            "<|reserved_special_token_4|>",
            "<|eot_id|>",  # end of turn
        ] + [f"<|reserved_special_token_{i}|>" for i in range(5, 256 - 5)]
mergeable_ranks = load_tiktoken_bpe(tokenizer_path)
tokenizer = tiktoken.Encoding(
    name=Path(tokenizer_path).name,
    pat_str=r"(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+",
    mergeable_ranks=mergeable_ranks,
    special_tokens={token: len(mergeable_ranks) + i for i, token in enumerate(special_tokens)},
)

tokenizer.decode(tokenizer.encode("hello world!"))

报错内容：

发现源头

通过email url的方式下载的模型权重和hf下载的差别很大

来源	文件
email
huggingface

如何获取Meta-Llama-3-8B

huggingface已有大佬将model搬运好，且不需要授权：bofenghuang/Meta-Llama-3-8B

重新运行

'hello world!'

参考

1. naklecha/llama3-from-scratch
2. Llama 3 使用方法以及模型下载教程
3. bofenghuang/Meta-Llama-3-8B