LEARNING NOTES FROM DEEP LEARNING

LEARNING NOTES FROM DEEP LEARNING

CHAPTER ONE         INTRODUCTION

        The note is about what I have learned in the first chapter. In this note, I will divide my learning into six parts: desires and AI solution, why AI? And what is its core-competitiveness, the history of AI’s development, the reality and AI’s contributions, connections between AI’s different subjects, main readers and the book structure. All the following things are based on what I had understood about the book. Therefore, if something goes wrong, you can comment on it and I am desired to correct it. Here we go!

        Actually,a long time ago, Pygmalion invented Galatea who was given life by goddess.Daedalus had invented lots of things to help him reach his ambitions.Hephaestus was the Vulcan who created the Talos and famous Pandora remembered by the offspring. They were seemed as the genius inventors who created real lives and until now, humans still desire to search for an engine to create them.Fortunately, that’s what AI can provide! Sometimes robot’s real problem was seemed so easy when it was asked to be done by humans such as the Recognition of human face. The same situations for humans when they need to compute amount of calculations exist as well. However, AI can use the hierarchy of concepts to understand the world as well as solving the difficulties above. On account of his learning ways based on the increasing complicated concepts as well as deeper understanding of the abstract information, we call its way ‘Deep Learning’.

        AI’s appearing is not accidental. In 1990s, IBM’s DeepBlue defeated the world’s most capable chess players for the rules are simple, robot can be programmed to calculate limited steps’ success rates. In other fields, robot still couldn’t recognize one’s face or had his own mind to ‘think’ freely. These failures could due to the hard-code knowledge’s failure. A significant example is Cyc’s,you can imagine a picture that a man is using his shaver, which is regarded as a machine by Cyc just because the shaver is a machine, HAHA~. Since the way humans teaching robots how to do by a list of rules (hard-code knowledge) didn’t work, scientists desired that robots can directly learn from the given dataset, which comes to the considerations of data’s representation. The represent form of knowledge can influence your learning ability to a great extent. When you hope to separate the blue dots and green nabla, you only achieve it in the right coordinates, but they share the same dots location, how it happens?That’s because we change coordinates to present the dots, which proves the significance of representation of knowledge, the same as the data’s representation.

        Therefore,as soon as you choose a suitable data’s representation can you consider the next step: to determine features of designated thing that need to give robot as the training set. Here scientists search for a way that can avoid the artificial choice of what features should be presented to the robot. The answer is ‘Representation Learning’, by which robot is able to learn the function from the original data and avoid time spending on it to select the features. The last question is that when we let robot learn from different pictures, information, signals, there will exist factors of variation from the noise, shadow or anything else. How can robot correctly recognize the real object by ignoring other influence factors?And that’s unique skill which deep learning can provide. The following picture show how deep learning AI obtain the aim by deep ‘hierarchy of concepts’.

In the first layer of the AI’s input, the robot can recognize and input system the single pixels, by which robot gets more information about the picture’s objects’ edges when getting more pixels. Then robot can get more abstract understanding about the picture’s objects. When these procedures loops, the last time robot can recognize the whole objects without difficulties. You can find there exist many layers, that’s the meaning of‘deep’ learning.

        Soon you will need solve another problem: How deep is the AI? There exists two popular ways: calculate these sequential instructions or different concepts relationships. The next picture shows the choice of different forms of represent the sequential instructions can lead to different result. If you choose the symbols element set, you will build four layers. But if you choose the method element set, you only build two layers. You can also calculate the relationships of invoking a variety of concepts during the building to know the depth. And that’s what AI could do to understand abstract things which appeals to many researchers or scientists.

Knowing the history of Deep Learning’s development can help us better understand how Deep Learning was created. In 1940s~1960s, scientists imitate biological brain ways to manufacture a machine,in this way they can somehow understand the brain functions and principles as well as prove that intelligent behavior is possible when duplicating it functionality. During this time, deep learning was called ‘Cybernetics’,fundamental methods Linear model and Stochastic gradient descend were invented comprehensively. Well, in the following development, Deep Learning’s research diminish the role of neuroscience impact because we didn’t grasp much knowledge about our brain and computational neuroscience is the main subject studying the function of one’s brain. Single neuron’s ability was limited when handling complicated problems. With the idea that a large number of simple computational units can achieve intelligent behavior when networked together, in 1970s~1980s, Connectionism was put forward and made many important advances in modeling sequences with neural network. Also, the distributed representation, back propagation, LSTM which still remain central to today’s deep learning were invented. However, the unrealistically ambitions claimed by AI technologies,other fields of machine learning’s achievements and incomplete hardware technology led to a decline in the popularity of neural networks until 2007.Thanks to the explosion of big data and booming of hardware technology based on the Moore's law, deep learning no longer need complicated well-designed structures to reach the high accuracy rate and scientists can focus more on the theories and the source of data set. Until now, many projects and internet companies used the deep learning to gain huge profits. Let’s see the reality and deep learning’s contributions nowadays.

        The increasing dataset sizes decline the use of techniques and simplify the structure in deep learning that scientists can pay more attention on the research of unsupervised learning. The increasing model sizes also save amount of computational resources which contributes to the increasing accuracy as well as huge impact towards real-world. Deep learning gain high profits and behave well in ImageNet, Speech recognition, Pedestrian detection and image segmentation, entire sequence of characters, neural Turing machines, reinforcement learning, etc. in virtue of the software infrastructure. The following pictures show the progress of deep learning:

        You may want to know the connections between different AI’s subjects, you can get understanding of it from the pictures below. You will find that we gradually give robots more freedom to let it behave well, at the same time we didn’t spend much time on training or supervising it.

Last but not least, this book’s main readers are for the University students,software engineers, etc. The book structure can be seen from the next pictures,the fields of Computer vision\speech and audio processing\ NLP \robotics\bioinformatics and chemistry \video games\search engine\online advertising and finance all are applied by the deep learning.