大语言模型生成式AI学习笔记——1.3.1 第1周参考资料

Week 1 resources（第1周参考资料）

Below you'll find links to the research papers discussed in this weeks videos. You don't need to understand all the technical details discussed in these papers - you have already seen the most important points you'll need to answer the quizzes in the lecture videos.

However, if you'd like to take a closer look at the original research, you can read the papers and articles via the links below.

1. Generative AI Lifecycle

• Generative AI on AWS: Building Context-Aware, Multimodal Reasoning Applications - This O'Reilly book dives deep into all phases of the generative AI lifecycle including model selection, fine-tuning, adapting, evaluation, deployment, and runtime optimizations.

1. Transformer Architecture

• Attention is All You Need - This paper introduced the Transformer architecture, with the core “self-attention” mechanism. This article was the foundation for LLMs.
• BLOOM: BigScience 176B Model - BLOOM is a open-source LLM with 176B parameters trained in an open and transparent way. In this paper, the authors present a detailed discussion of the dataset and process used to train the model. You can also see a high-level overview of the model here.
• Vector Space Models - Series of lessons from DeepLearning.AI's Natural Language Processing specialization discussing the basics of vector space models and their use in language modeling.

1. Pre-training and scaling laws

• Scaling Laws for Neural Language Models - empirical study by researchers at OpenAI exploring the scaling laws for large language models.

1. Model architectures and pre-training objectives

• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? - The paper examines modeling choices in large pre-trained language models and identifies the optimal approach for zero-shot generalization.
• HuggingFace Tasks and Model Hub - Collection of resources to tackle varying machine learning tasks using the HuggingFace library.
• LLaMA: Open and Efficient Foundation Language Models - Article from Meta AI proposing Efficient LLMs (their model with 13B parameters outperform GPT3 with 175B parameters on most benchmarks)

1. Scaling laws and compute-optimal models

• Language Models are Few-Shot Learners - This paper investigates the potential of few-shot learning in Large Language Models.
• Training Compute-Optimal Large Language Models - Study from DeepMind to evaluate the optimal model size and number of tokens for training LLMs. Also known as “Chinchilla Paper”.
• BloombergGPT: A Large Language Model for Finance - LLM trained specifically for the finance domain, a good example that tried to follow chinchilla laws.

在下面，您将找到本周视频中讨论的研究论文的链接。您不需要理解这些论文中讨论的所有技术细节 - 您已经看到了回答讲座视频中测验所需的最重要的要点。然而，如果您想更深入地了解原始研究，可以通过下面的链接阅读论文和文章。

1. 生成式AI生命周期

• 《AWS上的生成式AI：构建上下文感知的多模态推理应用》 - 这本O'Reilly书深入探讨了生成式AI生命周期的所有阶段，包括模型选择、微调、适应、评估、部署和运行时优化。

1. Transformer架构

• 《注意力是你所需要的一切》 - 这篇论文介绍了带有核心“自注意力”机制的Transformer架构。这篇文章是LLMs的基础。

 • 《BLOOM：BigScience 176B模型》 - BLOOM是一个开源的LLM，具有176B个参数，以开放透明的方式训练。在这篇论文中，作者详细讨论了用于训练模型的数据集和过程。您还可以在这里看到模型的高级概述。

 • 《向量空间模型》 - DeepLearning.AI的自然语言处理专业系列课程中的一序列课程，讨论了向量空间模型的基础知识及其在语言建模中的应用。

1. 预训练和缩放法则

• 《神经语言模型的缩放法则》 - OpenAI的研究人员进行的实证研究，探索了大型语言模型的缩放法则。

1. 模型架构和预训练目标

•《什么语言模型架构和预训练目标最适合零射击泛化？》 - 该论文检查了大型预训练语言模型中的建模选择，并确定了零射击泛化的最优方法。

• 《HuggingFace任务和模型中心》 - 使用HuggingFace库解决不同机器学习任务的资源集合。

• 《LLaMA：开放高效的基础语言模型》 - Meta AI提出的文章，提出高效的LLMs（他们的模型具有13B个参数，在大多数基准测试中优于具有175B个参数的GPT3）。

1. 缩放法则和计算优化模型

• 《语言模型是少样本学习者》 - 这篇论文调查了大型语言模型中少样本学习的潜力。

 • 《训练计算优化的大型语言模型》 - DeepMind的研究，评估训练LLMs的最佳模型大小和令牌数量。也被称为“Chinchilla论文”。

• 《BloombergGPT：金融领域的大型语言模型》 - 专门为金融领域训练的LLM，是一个很好的例子，尝试遵循chinchilla法则。