大语言模型（LLM）的进化树，学习LLM看明白这一张图就够了

近期大语言模型迅速发展，让大家看得眼花缭乱，感觉现在LLM的快速发展堪比寒武纪大爆炸，各个模型之间的关系也让人看的云里雾里。最近一些学者整理出了 ChatGPT 等语言模型的发展历程的进化树图，让大家可以对LLM之间的关系一目了然。

最重要的进化树图：

进化的树图

现代语言模型的进化树追溯了近年来语言模型的发展,并强调了一些最著名的模型。同一分支上的模型关系更近。基于Transformer的模型显示为非灰色颜色:仅解码器模型显示为蓝色分支,仅编码器模型显示为粉红色分支,而编码器-解码器模型显示为绿色分支。时间轴上模型的垂直位置代表其发布日期。开源模型由实心方块表示,而闭源模型由空心方块表示。右下角的堆叠条形图显示来自各公司和机构的模型数量。

然后是按年进化的动图，主要内容和上图相同。

论文内容简介（Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond）

趋势

a) 仅解码器模型在语言模型的发展中逐渐占主导地位。在语言模型发展的早期阶段，仅解码器模型不如仅编码器模型和编码器-解码器模型流行。然而，在2021年之后，随着改变游戏规则的语言模型GPT-3的推出，仅解码器模型经历了显著的繁荣。与此同时，在BERT带来的初始爆炸性增长之后，仅编码器模型逐渐开始消失。

b) OpenAI在语言模型领域始终保持领先地位，无论当前还是未来。其他公司和机构正在努力追赶OpenAI开发可与GPT-3和当前的GPT-4相媲美的模型。这一领先地位可归因于OpenAI即使在最初并不广泛认可的情况下也坚定地坚持其技术路线。

c) Meta对开源语言模型做出了重大贡献，并促进了语言模型的研究。在考虑对开源社区的贡献时，特别是与语言模型相关的贡献，Meta作为最慷慨的商业公司之一脱颖而出，因为Meta开发的所有语言模型都是开源的。

d) 语言模型表现出倾向于闭源的趋势。在语言模型发展的早期阶段(2020年之前)，大多数模型都是开源的。然而，随着GPT-3的推出，公司越来越倾向于闭源其模型，如PaLM、LaMDA和GPT-4。因此，学术研究人员更难进行语言模型训练的实验。因此，基于API的研究可能成为学术界的主流方法。

e) 编码器-解码器模型仍然具有前景，因为这种体系结构仍在积极探索中，并且大多数都是开源的。Google对开源编码器-解码器架构做出了重大贡献。然而，仅解码器模型的灵活性和多功能性似乎使得Google坚持这一方向的前景不太乐观。

总之，仅解码器模型和开源模型在近年来占据主导地位，而OpenAI和Meta在推动语言模型创新和开源方面做出了重大贡献。与此同时，编码器-解码器模型和闭源模型也在一定程度上推动了发展。各家公司和机构在技术发展路径上面临不同的前景。

模型实用指南（Practical Guide for Models）

LLM实用指南资源的精选（仍在积极更新）列表。它基于调查论文：在实践中利用LLM的力量：关于ChatGPT及其他的调查。这些资源旨在帮助从业者驾驭大型语言模型（LLM）及其在自然语言处理（NLP）应用程序中的应用。

BERT风格的语言模型：编码器-解码器或仅编码器（Encoder-Decoder or Encoder-only）

• BERT BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018, Paper
• RoBERTa ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, 2019, Paper
• DistilBERT DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, 2019, Paper
• ALBERT ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, 2019, Paper
• UniLM Unified Language Model Pre-training for Natural Language Understanding and Generation, 2019 Paper
• ELECTRA ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS, 2020, Paper
• T5 “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”. Colin Raffel et al. JMLR 2019. Paper
• GLM “GLM-130B: An Open Bilingual Pre-trained Model”. 2022. Paper
• AlexaTM “AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model”. Saleh Soltan et al. arXiv 2022. Paper
• ST-MoE ST-MoE: Designing Stable and Transferable Sparse Expert Models. 2022 Paper

GPT 风格语言模型：仅解码器（Decoder-only）

• GPT Improving Language Understanding by Generative Pre-Training. 2018. Paper
• GPT-2 Language Models are Unsupervised Multitask Learners. 2018. Paper
• GPT-3 “Language Models are Few-Shot Learners”. NeurIPS 2020. Paper
• OPT “OPT: Open Pre-trained Transformer Language Models”. 2022. Paper
• PaLM “PaLM: Scaling Language Modeling with Pathways”. Aakanksha Chowdhery et al. arXiv 2022. Paper
• BLOOM “BLOOM: A 176B-Parameter Open-Access Multilingual Language Model”. 2022. Paper
• MT-NLG “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”. 2021. Paper
• GLaM “GLaM: Efficient Scaling of Language Models with Mixture-of-Experts”. ICML 2022. Paper
• Gopher “Scaling Language Models: Methods, Analysis & Insights from Training Gopher”. 2021. Paper
• chinchilla “Training Compute-Optimal Large Language Models”. 2022. Paper
• LaMDA “LaMDA: Language Models for Dialog Applications”. 2021. Paper
• LLaMA “LLaMA: Open and Efficient Foundation Language Models”. 2023. Paper
• GPT-4 “GPT-4 Technical Report”. 2023. Paper
• BloombergGPT BloombergGPT: A Large Language Model for Finance, 2023, Paper
• GPT-NeoX-20B: “GPT-NeoX-20B: An Open-Source Autoregressive Language Model”. 2022. Paper

数据实用指南

预训练数据

• RedPajama, 2023. Repo
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling, Arxiv 2020. Paper
• How does the pre-training objective affect what large language models learn about linguistic properties?, ACL 2022. Paper
• Scaling laws for neural language models, 2020. Paper
• Data-centric artificial intelligence: A survey, 2023. Paper
• How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources, 2022. Blog

微调数据

• Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach, EMNLP 2019. Paper
• Language Models are Few-Shot Learners, NIPS 2020. Paper
• Does Synthetic Data Generation of LLMs Help Clinical Text Mining? Arxiv 2023 Paper

测试数据/用户数据

• Shortcut learning of large language models in natural language understanding: A survey, Arxiv 2023. Paper
• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective Arxiv, 2023. Paper
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems Arxiv 2019. Paper

NLP 任务实用指南

研究者为用户的NLP应用程序构建了一个选择LLM或微调模型的决策流程~\protect\footnotemark。决策流程可帮助用户评估其手头的下游NLP应用程序是否满足特定条件，并根据该评估确定LLM或微调模型是否最适合其应用程序。

传统的非语言处理任务（ NLU tasks)

• A benchmark for toxic comment classification on civil comments dataset Arxiv 2023 Paper
• Is chatgpt a general-purpose natural language processing task solver? Arxiv 2023Paper
• Benchmarking large language models for news summarization Arxiv 2022 Paper

生成任务

• News summarization and evaluation in the era of gpt-3 Arxiv 2022 Paper
• Is chatgpt a good translator? yes with gpt-4 as the engine Arxiv 2023 Paper
• Multilingual machine translation systems from Microsoft for WMT21 shared task, WMT2021 Paper
• Can ChatGPT understand too? a comparative study on chatgpt and fine-tuned bert, Arxiv 2023, Paper

知识密集型任务

• Measuring massive multitask language understanding, ICLR 2021 Paper
• Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, Arxiv 2022 Paper
• Inverse scaling prize, 2022 Link
• Atlas: Few-shot Learning with Retrieval Augmented Language Models, Arxiv 2022 Paper
• Large Language Models Encode Clinical Knowledge, Arxiv 2022 Paper

缩放能力

• Training Compute-Optimal Large Language Models, NeurIPS 2022 Paper
• Scaling Laws for Neural Language Models, Arxiv 2020 Paper
• Solving math word problems with process- and outcome-based feedback, Arxiv 2022 Paper
• Chain of thought prompting elicits reasoning in large language models, NeurIPS 2022 Paper
• Emergent abilities of large language models, TMLR 2022 Paper
• Inverse scaling can become U-shaped, Arxiv 2022 Paper
• Towards Reasoning in Large Language Models: A Survey, Arxiv 2022 Paper

具体任务

• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks, Arixv 2022 Paper
• PaLI: A Jointly-Scaled Multilingual Language-Image Model, Arxiv 2022 Paper
• AugGPT: Leveraging ChatGPT for Text Data Augmentation, Arxiv 2023 Paper
• Is gpt-3 a good data annotator?, Arxiv 2022 Paper
• Want To Reduce Labeling Cost? GPT-3 Can Help, EMNLP findings 2021 Paper
• GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation, EMNLP findings 2021 Paper
• LLM for Patient-Trial Matching: Privacy-Aware Data Augmentation Towards Better Performance and Generalizability, Arxiv 2023 Paper
• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks, Arxiv 2023 Paper
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, Arxiv 2023 Paper
• GPTScore: Evaluate as You Desire, Arxiv 2023 Paper
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality, Arxiv 2023 Paper
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study, Arxiv 2023 Paper

现实世界的“任务”

• Sparks of Artificial General Intelligence: Early experiments with GPT-4, Arxiv 2023 Paper

效率

1.成本

• Openai’s gpt-3 language model: A technical overview, 2020. Blog Post
• Measuring the carbon intensity of ai in cloud instances, FaccT 2022. Paper
• In AI, is bigger always better?, Nature Article 2023. Article
• Language Models are Few-Shot Learners, NeurIPS 2020. Paper
• Pricing, OpenAI. Blog Post

2.延迟

• HELM: Holistic evaluation of language models, Arxiv 2022. Paper

3.参数高效微调

• LoRA: Low-Rank Adaptation of Large Language Models, Arxiv 2021. Paper
• Prefix-Tuning: Optimizing Continuous Prompts for Generation, ACL 2021. Paper
• P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks, ACL 2022. Paper
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks, Arxiv 2022. Paper

4.预训练系统

• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, Arxiv 2019. Paper
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, Arxiv 2019. Paper
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, Arxiv 2021. Paper
• Reducing Activation Recomputation in Large Transformer Models, Arxiv 2021. Paper

信用

1. 稳健性和校准

• Calibrate before use: Improving few-shot performance of language models, ICML 2021. Paper
• SPeC: A Soft Prompt-Based Calibration on Mitigating Performance Variability in Clinical Notes Summarization, Arxiv 2023. Paper

2. 杂散偏置（Spurious biases）

• Shortcut learning of large language models in natural language understanding: A survey, 2023 Paper
• Mitigating gender bias in captioning system, WWW 2020 Paper
• Calibrate Before Use: Improving Few-Shot Performance of Language Models, ICML 2021 Paper
• Shortcut Learning in Deep Neural Networks, Nature Machine Intelligence 2020 Paper
• Do Prompt-Based Models Really Understand the Meaning of Their Prompts?, NAACL 2022 Paper

3. 安全问题

• GPT-4 System Card, 2023 Paper
• The science of detecting llm-generated texts, Arxiv 2023 Paper
• How stereotypes are shared through language: a review and introduction of the aocial categories and stereotypes communication (scsc) framework, Review of Communication Research, 2019 Paper
• Gender shades: Intersectional accuracy disparities in commercial gender classification, FaccT 2018 Paper

基准指导调优（Benchmark Instruction Tuning）

• FLAN: Finetuned Language Models Are Zero-Shot Learners, Arxiv 2021 Paper
• T0: Multitask Prompted Training Enables Zero-Shot Task Generalization, Arxiv 2021 Paper
• Cross-task generalization via natural language crowdsourcing instructions, ACL 2022 Paper
• Tk-INSTRUCT: Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks, EMNLP 2022 Paper
• FLAN-T5/PaLM: Scaling Instruction-Finetuned Language Models, Arxiv 2022 Paper
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning, Arxiv 2023 Paper
• OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization, Arxiv 2023 Paper

校准（Alignment）

• Deep Reinforcement Learning from Human Preferences, NIPS 2017 Paper
• Learning to summarize from human feedback, Arxiv 2020 Paper
• A General Language Assistant as a Laboratory for Alignment, Arxiv 2021 Paper
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Arxiv 2022 Paper
• Teaching language models to support answers with verified quotes, Arxiv 2022 Paper
• InstructGPT: Training language models to follow instructions with human feedback, Arxiv 2022 Paper
• Improving alignment of dialogue agents via targeted human judgements, Arxiv 2022 Paper
• Scaling Laws for Reward Model Overoptimization, Arxiv 2022 Paper
• Scalable Oversight: Measuring Progress on Scalable Oversight for Large Language Models, Arxiv 2022 Paper

安全校准（无害）

• Red Teaming Language Models with Language Models, Arxiv 2022 Paper
• Constitutional ai: Harmlessness from ai feedback, Arxiv 2022 Paper
• The Capacity for Moral Self-Correction in Large Language Models, Arxiv 2023 Paper
• OpenAI: Our approach to AI safety, 2023 Blog

真实性一致性（诚实）

• Reinforcement Learning for Language Models, 2023 Blog

提示实用指南（有用）

• OpenAI Cookbook. Blog
• Prompt Engineering. Blog
• ChatGPT Prompt Engineering for Developers!Course

开源社区的工作

• Self-Instruct: Aligning Language Model with Self Generated Instructions, Arxiv 2022 Paper
• Alpaca. Repo
• Vicuna. Repo
• Dolly. Blog
• DeepSpeed-Chat. Blog
• GPT4All. Repo
• OpenAssitant. Repo
• ChatGLM. Repo
• MOSS. Repo
• Lamini. Repo/Blog

如何系统的去学习大模型LLM ？

作为一名热心肠的互联网老兵，我意识到有很多经验和知识值得分享给大家，也可以通过我们的能力和经验解答大家在人工智能学习中的很多困惑，所以在工作繁忙的情况下还是坚持各种整理和分享。

但苦于知识传播途径有限，很多互联网行业朋友无法获得正确的资料得到学习提升，故此将并将重要的 AI大模型资料 包括AI大模型入门学习思维导图、精品AI大模型学习书籍手册、视频教程、实战学习等录播视频免费分享出来。

😝有需要的小伙伴，可以V扫描下方二维码免费领取🆓

一、全套AGI大模型学习路线

AI大模型时代的学习之旅：从基础到前沿，掌握人工智能的核心技能！

二、640套AI大模型报告合集

这套包含640份报告的合集，涵盖了AI大模型的理论研究、技术实现、行业应用等多个方面。无论您是科研人员、工程师，还是对AI大模型感兴趣的爱好者，这套报告合集都将为您提供宝贵的信息和启示。

三、AI大模型经典PDF籍

随着人工智能技术的飞速发展，AI大模型已经成为了当今科技领域的一大热点。这些大型预训练模型，如GPT-3、BERT、XLNet等，以其强大的语言理解和生成能力，正在改变我们对人工智能的认识。 那以下这些PDF籍就是非常不错的学习资源。

四、AI大模型商业化落地方案

阶段1：AI大模型时代的基础理解

• 目标：了解AI大模型的基本概念、发展历程和核心原理。
• 内容：
  • L1.1 人工智能简述与大模型起源
  • L1.2 大模型与通用人工智能
  • L1.3 GPT模型的发展历程
  • L1.4 模型工程
    - L1.4.1 知识大模型
    - L1.4.2 生产大模型
    - L1.4.3 模型工程方法论
    - L1.4.4 模型工程实践
  • L1.5 GPT应用案例

阶段2：AI大模型API应用开发工程

• 目标：掌握AI大模型API的使用和开发，以及相关的编程技能。
• 内容：
  • L2.1 API接口
    - L2.1.1 OpenAI API接口
    - L2.1.2 Python接口接入
    - L2.1.3 BOT工具类框架
    - L2.1.4 代码示例
  • L2.2 Prompt框架
    - L2.2.1 什么是Prompt
    - L2.2.2 Prompt框架应用现状
    - L2.2.3 基于GPTAS的Prompt框架
    - L2.2.4 Prompt框架与Thought
    - L2.2.5 Prompt框架与提示词
  • L2.3 流水线工程
    - L2.3.1 流水线工程的概念
    - L2.3.2 流水线工程的优点
    - L2.3.3 流水线工程的应用
  • L2.4 总结与展望

阶段3：AI大模型应用架构实践

• 目标：深入理解AI大模型的应用架构，并能够进行私有化部署。
• 内容：
  • L3.1 Agent模型框架
    - L3.1.1 Agent模型框架的设计理念
    - L3.1.2 Agent模型框架的核心组件
    - L3.1.3 Agent模型框架的实现细节
  • L3.2 MetaGPT
    - L3.2.1 MetaGPT的基本概念
    - L3.2.2 MetaGPT的工作原理
    - L3.2.3 MetaGPT的应用场景
  • L3.3 ChatGLM
    - L3.3.1 ChatGLM的特点
    - L3.3.2 ChatGLM的开发环境
    - L3.3.3 ChatGLM的使用示例
  • L3.4 LLAMA
    - L3.4.1 LLAMA的特点
    - L3.4.2 LLAMA的开发环境
    - L3.4.3 LLAMA的使用示例
  • L3.5 其他大模型介绍

阶段4：AI大模型私有化部署

• 目标：掌握多种AI大模型的私有化部署，包括多模态和特定领域模型。
• 内容：
  • L4.1 模型私有化部署概述
  • L4.2 模型私有化部署的关键技术
  • L4.3 模型私有化部署的实施步骤
  • L4.4 模型私有化部署的应用场景

学习计划：

• 阶段1：1-2个月，建立AI大模型的基础知识体系。
• 阶段2：2-3个月，专注于API应用开发能力的提升。
• 阶段3：3-4个月，深入实践AI大模型的应用架构和私有化部署。
• 阶段4：4-5个月，专注于高级模型的应用和部署。

这份完整版的大模型 LLM 学习资料已经上传CSDN，朋友们如果需要可以微信扫描下方CSDN官方认证二维码免费领取【保证100%免费】

😝有需要的小伙伴，可以Vx扫描下方二维码免费领取🆓