Megatron-LM运行错误

u013250861

已于 2024-02-14 13:13:23 修改

阅读量403

点赞数 9

分类专栏：图神经网络 # LLM/预训练&SFT 文章标签：深度学习人工智能机器学习

于 2024-01-08 01:51:55 首次发布

本文链接：https://blog.csdn.net/u013250861/article/details/135447699

版权

LLM/预训练&SFT 同时被 2 个专栏收录

84 篇文章 81 订阅 ¥19.90 ¥99.00

订阅专栏

超级会员免费看

图神经网络

39 篇文章 16 订阅 ¥15.90 ¥99.00

订阅专栏

超级会员免费看

在深入探讨Megatron-LM模型时，遇到了内存效率问题。在`megatron/model/fused_layer_norm.py`的`MixedFusedLayerNorm`类中，通过将`memory_efficient`参数设置为True，可以优化`apex/contrib/layer_norm/layer_norm.py`中`FastLayerNormFN`类的`forward`方法，以提升模型运行时的内存利用率。

摘要由CSDN通过智能技术生成

一、memory_efficient

> finished creating GPT datasets ...
[after dataloaders are built] datetime: 2024-01-07 17:48:27 
done with setup ...
(min, max) time across ranks (ms):
    model-and-optimizer-setup ......................: (850.00, 850.00)
    train/valid/test-data-iterators-setup ..........: (144.00, 144.00)
training ...
[before the start of training step] datetime: 2024-01-07 17:48:27 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++&#