Model architecture
standard decoder-only transformer
Compared to Llama 2, we made several key improvements:
- Llama 3 uses a tokenizer with a vocabulary of 128K tokens that encodes language much more efficiently
- To improve the inference efficiency of Llama 3 models, we’ve adopted grouped query attention (GQA) across both the 8B and 70B sizes
- We trained the models on sequences of 8,192 tokens, using a mask to ensure self-attention does not cross document boundaries.
Instruction fine-tuning
- supervised fine-tuning (SFT)
- rejection sampling
- proximal policy optimization (PPO)
- direct preference optimization (DPO):the preference rankings that are used in PPO and DPO has an outsized influence on the performance of aligned models