Bengio <Deep Learning> 笔记

书本介绍：The Deep Learning textbook is a resource intended to help students and practitioners enter the field of machine learning in general and deep learning in particular. The online version of the book is now complete and will remain available online for free. 

网址：http://www.deeplearningbook.org/

开始读了下Bengio的大作<Deep Learning>，把一些自己感觉有意思的文段摘抄下来并写写自己的体会。

Introduction

1. The true challenge to artificial intelligence proved to be solving the tasks that are easy for people to perform but hard for people to describe formally—problems that we solve intuitively, that feel automatic, like recognizing spoken words or faces in images.

计算机所擅长的就是按照我们所设定的执行路线去解决一个定义好的问题。但是很多习以为常的事情，我们却很难以进行形式化的描述。比如翻译问题，我们自己都没有搞清楚翻译过程大脑是怎么进行操作的。是一个词语一个词语的对着翻译，还是一个短语一个短语的对着翻译还是看完整个句子后在翻译？类似，人工智能中的很多问题不是简单的计算问题，更多涉及人类的感知、学习、理解和认知。对这些问题我们自己都没摸清楚，哪能写程序让计算机去弄清楚.......

2. Media accounts often emphasize the similarity of deep learning to the brain. While it is true that deep learning researchers are more likely to cite the brain as an influence than researchers working in other machine learning fields such as kernel machines or Bayesian statistics, one should not view deep learning as an attempt to simulate the brain. Modern deep learning draws inspiration from many fields, especially applied math fundamentals like linear algebra, probability, information theory, and numerical optimization. While some deep learning researchers cite neuroscience as an important source of inspiration, others are not concerned with neuroscience at all.

深度学习和大脑的关系。（很多讲深度学习的PPT中，至少要贴一个布满神经的大脑图。）

3. Several key concepts arose during the connectionism movement of the 1980s that remain central to today’s deep learning. 

（distributed representation + back-propagation algorithm）

distributed representation这想法原来早在三四十前面就玩过了。科研就是在前人基础上不断继续挖坑。

4. Ventures based on neural networks and other AI technologies began to make unrealistically ambitious claims while seeking investments. When AI research did not fulfill these unreasonable expectations, investors were disappointed.

当年火的一塌糊涂的神经网络就是由于吹破天害惨自己，难怪以Yann LeCun为首的一些研究员呼吁大家冷静对待这一波深度学习的热潮。

Deep FeedForward Networks

看到Bernoulli Distribution，竟然半天记不起来，基础不扎实。

伯努利分布、二项分布、多项分布、Beta分布、Dirichlet分布 http://blog.csdn.net/michael_r_chang/article/details/39188321