文章目录
- 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
- 02:21:06 learning rate scheduler: warmup + cosine decay
- 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
- 02:34:09 gradient accumulation【实现更大的batch】
- 02:46:52 distributed data parallel (DDP)【多gpu训练,想象8个节点同时运行一段逻辑,而不同的只有ddp_rank, 你必须仔细考虑代码逻辑!8份!】
- 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
- 03:23:10 validation data split, validation loss, sampling revive
from视频:https://www.youtube.com/watch?v=l8pRSuU81PU&t=8098s
简介
468,654次观看 2024年6月10日
We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.
02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
02:21:06 learning rate scheduler: warmup + cosine decay
02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
02:34:09 gradient accumulation
02:46:52 distributed data parallel (DDP)
03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
03:23:10 validation data split, validation loss, sampling revive
03:28:23 evaluation: HellaSwag, starting the run
02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
gradient clipping
02:21:06 learning rate scheduler: warmup + cosine decay
手写 学习率 改变函数
02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
听Andrej Karpathy讲 优化过程,有点哲学!
02:34:09 gradient accumulation【实现更大的batch】
02:46:52 distributed data parallel (DDP)【多gpu训练,想象8个节点同时运行一段逻辑,而不同的只有ddp_rank, 你必须仔细考虑代码逻辑!8份!】
初始化:你必须考虑同时8个节点 运行这一段逻辑!
8个节点拥有的不同,只是ddp_rank
调试结果
改变 dataloader
记住:每个进程知道的只有 ddp_rank,ddp_world_size, ddp_rank==0是主进程!
继续执行代码逻辑!
文章目录
- 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
- 02:21:06 learning rate scheduler: warmup + cosine decay
- 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
- 02:34:09 gradient accumulation【实现更大的batch】
- 02:46:52 distributed data parallel (DDP)【多gpu训练,想象8个节点同时运行一段逻辑,而不同的只有ddp_rank, 你必须仔细考虑代码逻辑!8份!】
- 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
- 03:23:10 validation data split, validation loss, sampling revive
gpu ddp全用上就是快,1.5 million tokens / sec
03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
分块保存数据
03:23:10 validation data split, validation loss, sampling revive
采样,模型输出
新的评估方式