低资源如何用深度模型？【2022新书】高效深度学习: 更快更小更好的模型，Efficient Deep Learning全面解答...

来源：专知
本文约1000字，建议阅读5分钟
在这本书中，我们提出了一个基本的平衡，以及实践知识，以充分赋能你继续前进。

深度学习领域取得了指数级的发展，像BERT、GPT-3、ResNet等ML模型的足迹也在不断扩大。虽然它们工作得很好，但在生产中训练和部署这些大型(且不断增长的)模型是昂贵的。你可能想在智能手机上部署你的面部滤镜模型，让你的用户在他们的自拍上添加一个小狗滤镜。但它可能太大或太慢，或者您可能想提高基于云的垃圾邮件检测模型的质量，但又不想花钱购买更大的云VM来承载更精确但更大的模型。如果您的模型没有足够的标记数据，或者不能手动调优您的模型，该怎么办? 所有这些都是令人生畏的!

如果您可以使您的模型更高效: 使用更少的资源(模型大小、延迟、训练时间、数据、人工参与)，并提供更好的质量(准确性、精确度、召回等)，会怎么样呢?这听起来太棒了! 但如何?

这本书将通过在谷歌研究，Facebook人工智能研究(FAIR)，和其他著名的人工智能实验室使用算法和技术的研究人员和工程师训练和部署他们的模型，设备从大型服务器端机器到微型微控制器。在这本书中，我们提出了一个基本的平衡，以及实践知识，以充分赋能你继续前进，并优化你的模型训练和部署工作流，这样你的模型表现和以前一样好或更好，与一小部分资源。我们还将深入介绍流行的模型、基础设施和硬件，以及具有挑战性的项目，以测试您的技能。

https://efficientdlbook.com/

目录内容：

Part I: 高效深度学习导论 Introduction to Efficient Deep Learning

1. 导论 Introduction

• Introduction to Deep Learning

• Efficient Deep Learning

• Mental Model of Efficient Deep Learning

• Summary

Part II: Effciency Techniques

1. 压缩技术导论 Introduction to Compression Techniques

• An Overview of Compression

• Quantization

• Exercises: Compressing images from the Mars Rover

• Project: Quantizing a Deep Learning Model

• Summary

学习技术导论 Introduction to Learning Techniques

• Project: Increasing the accuracy of an speech identification model with Distillation.

• Project: Increasing the accuracy of an image classification model with Data Augmentation.

• Project: Increasing the accuracy of a text classification model with Data Augmentation.

• Learning Techniques and Efficiency

• Data Augmentation

• Distillation

• Summary

高效架构 Efficient Architectures

• Project: Project: Snapchat-Like Filters for Pets

• Project: News Classification Using RNN and Attention Models

• Project: Using pre-trained embeddings to improve accuracy of a NLP task.

• Embeddings for Smaller and Faster Models

• Learn Long-Term Dependencies Using Attention

• Efficient On-Device Convolutions

• Summary

高级压缩技术 Advanced Compression Techniques

• Exercise: Using clustering to compress a 1-D tensor.

• Exercise: Mars Rover beckons again! Can we do better with clustering?

• Exercise: Simulating clustering on a dummy dense fully-connected layer

• Project: Using Clustering to compress a deep learning model

• Exercise: Sparsity improves compression

• Project: Lightweight model for pet filters application

• Model Compression Using Sparsity

• Weight Sharing using Clustering

• Summary

高级学习技术 Advanced Learning Techniques

• Contrastive Learning

• Unsupervised Pre-Training

• Project: Learning to classify with 10% labels.

• Curriculum Learning

自动化 Automation

• Project: Layer-wise Sparsity to achieve a pareto optimal model.

• Project: Searching over model architectures for boosting model accuracy.

• Project: Multi-objective tuning to get a smaller and more accurate model.

• Hyper-Parameter Tuning

• AutoML

• Compression Search

Part 3 - Infrastructure

1. 软件基础 Software Infrastructure

• PyTorch Ecosystem

• iOS Ecosystem

• Cloud Ecosystems

硬件基础 Hardware infrastructure

• GPUs

• Jetson

• TPU

• M1 / A4/5?

• Microcontrollers

Part 3 - Applied Deep Dives

1. Deep-Dives: Tensorflow Platforms

• Project: Training BERT efficiently with TPUs.

• Project: Face recognition on the web with TensorFlow.JS.

• Project: Speech detection on a microcontroller with TFMicro.

• Project: Benchmarking a tiny on-device model with TFLite.

• Mobile

• Microcontrollers

• Web

• Google Tensor Processing Unit (TPU)

• Summary

Deep-Dives: Efficient Models

• Project: Efficient speech detection models.

• Project: Comparing efficient mobile models on Mobile.

• Project: Training efficient BERT models.

• BERT

• MobileNet

• EfficientNet architectures

• Speech Detection