阿正的梦工坊

Use sum for tasks that prioritize long-sequence performance, such as chat models or long-text generation.

finetune.py 脚本支持从 checkpoint 恢复训练，并且在加载模型权重时还会恢复训练的步数（steps） 和 epoch 信息

根据 transformers 文档，如果没有显式指定 label_pad_token_id，它通常会默认使用 -100，因为这是 CrossEntropyLoss 的 ignore_index 默认值。

统计tokenize之后长度超过 1024 的样本数量

Useful if tokenization process is long. Default is 1800 seconds (30 minutes)

In-Depth Analysis of Memory Requirements for Mixed Precision Training of Meta-Llama-3-8B-Instruct Model

混合精度训练通过 BF16 格式大幅减少显存需求，但关键的优化器参数（权重更新副本、一阶动量、二阶动量）仍然以 FP32 存储，保证数值稳定性和训练精度

In deep learning model training, a "step" refers to a single update of the model's parameters after processing a batch of training samples.

监督微调（Supervised Fine-Tuning, SFT）是指在一个已经经过预训练的大规模语言模型的基础上，使用标注数据进行进一步的训练，使其在某个特定任务上表现得更好。

在 SFT训练中，我们通常会使用有标签的训练数据进行微调。

有参模型通过有限的参数数量来定义模型，而非参模型通过数据本身或无固定数量的参数来描述问题。

Inference in large language models (LLMs), such as LLaMA 2 7B, involves two primary components of GPU memory consumption: model parameter memory and activation memory

A Detailed Analysis of SGD, RMSProp, and Adam Optimizers, and Their Memory Consumption

Both aim to improve training efficiency by adapting learning rates for each parameter, but they do so in different ways.

Detailed Analysis of Adam and AdamW Optimizers and Their Memory Consumption with float32 and bfloat16 Precision

Understanding the Additional Memory Requirements of Adam Optimizer: Memory Consumption Breakdown for Model Parameters and Optimizer States

BF16 offers a larger numerical range and is specifically optimized for deep learning tasks that require handling large gradients and weights.

Data Parallelism, Model Parallelism, and Tensor Parallelism: Parallel Computing Strategies in Deep Learning

This parameter allows users to manage and optimize the storage and version control of training states, facilitating efficient data storage and recovery during model training.

这些配置项涵盖了内存优化、自动调优、混合精度、分布式训练等多个方面，以及模型训练的其他细节方面，包括压缩、梯度处理、优化器配置、数据效率、流水线并行等

reduce_bucket_size is an essential parameter in DeepSpeed's ZeRO Stage 2 optimization, controlling the size of the buckets during gradient reduction.

By understanding the mechanics of gradient reduction and the impact of contiguous memory, we can optimize distributed training setups and improve model training efficiency across multiple devices.

dataset_info.json文件记得更改

It determines the maximum input sequence length (in tokens) that the model can process in a single forward pass after tokenization.

It controls whether the gradients are stored in a contiguous memory block.

DeepSpeed’s configuration is highly flexible, but tuning requires balancing memory efficiency and computational speed.

NCCL (NVIDIA Collective Communications Library) is a high-performance communication library developed by NVIDIA.

DeepSpeed introduces the ZeRO (Zero Redundancy Optimizer) optimization technique, a groundbreaking solution to reduce memory usage and improve efficiency during training.

训模型的经验

Pretraining involves teaching an LLM to predict the next token using vast text datasets, resulting in a base model, and this base model requires further fine-tuning for optimal performance and safety.

Write advanced Cypher queries to retrieve relevant information from the graph and format it for inclusion in your prompt to an LLM.

Quantization techniques

Build and customize multi-agent systems

The Amazon Titan model

Multiple data modalities

Mistral

Gradio Framework

AI Hub Models of Qualcomm

CrewAI for multi agents

DeepLearning.AI的short course