Introduction to Deep Learning - Supervised Learning with Neural Networks

Notes when studying Neural Networks & Deep Learning by Andrew Ng.

---

It turns out that so far almost all the economic value created by neural networks has been through supervised learning. Let's see what that means and we'll go through some examples in this class.

Figure-1

In supervised learning, you have some input x, and you want to learn a function mapping to some output y. In figure-1, besides the housing price prediction, we have more examples:

• Possibly, the single most lucrative application of deep learning today is online advertising. By inputting the information of an ad and some information about the user, neural networks have got very good at predicting whether or not you'll click on the ad. And by showing you the Ads that you're most likely to click on, this has been an incredibly lucrative application of neural networks at multiple companies.
• Computer vision has also made huge strides in the last several years mostly due to deep learning. You might input an image and want to output an index say from 1 to 1,000 trying to tell you if this picture might be say one of a 1,000 differet images. So you might use that for photo tagging.
• The recent progress in speech recognition has been very exciting. You can now input an audio clip to a neural network, and have it output a text transcript.
• Machine translation has also made huge strides. Now you can have a neural network, input an English sentence and directly output say a Chinese sentense
• In autonomous driving, you might input an image, say a picture of what's in front of your car, as well as some information from a radar, and based on that, maybe a neural network can be trained to tell you the position of other cars on the road. This becomes a key component in autonomous driving systems.

So, a lot of the value creation through the neural networks has been through cleverly selecting what should be x and what should be y for your particular problem. And then fitting this supervised learning component into often a bigger system such as an autonomous vehicle.

It turns out that slightly different types of neural networks are useful for different applications, for example:

• For the real estate application and online Advertising, we use a universally standard neural network architecture
• For image applications, we'll often use Convolutional Neural Networks, often abbreviated CNN
• For sequence data like audio which has a temporal component and is most naturally represented as a one-dimensional time series, we often use RNN, a Recurrent Neural Network
• Languages, English and Chinese, the alphabets or the words come one at a time. So language is also most naturally represented as sequence data. So more complex version of RNNs are often used for these application
• For more complex applications like autonomous driving, where you have an image, that might suggest more of a CNN structure. And radar info which is something quite different. So you might end up with a more custom, or some more complex hybrid neural network architecture.

Figure-2

Just to be a bit more concrete about what are the standard NN, CNN and RNN architectures. In literatures, you might have been seen pictures like what shown in figure-2.

Figure-3

You might have also heard about applications of machine learning to both Structured Data and Unstructed Data. Structured Data means basically databases of data. For example, in housing price prediction, you might have a database or the column that tells you the size and the number of bedrooms. Or in predicting whether or not a user will click on an Ad, you might have the information about the user, such as the age, some information about the Ad, and then labels y you're trying to predict. In contrast, unstructured data refers to things like audio, images where you want to recognize what's in the image or text. Here the features might be the pixel values in the image or the individual words in a piece of text. Historically, it was much harder for computers to make sense of unstructured data compared to structured data. Thanks to deep learning and neural networks, computers are now much better at interpreting unstructured data as well compared to just a few years ago. And this creates opportunities for many new exciting applications that use speech recognition, image recognition, natural language processing on text, much more than was possible even just two or three years ago. In this course, a lot of the techniques we'll go over will apply to both structured data and to unstructured data.  For the purposes of explaining the algorithms, we'll draw a little bit more on examples that use unstructured data. But as you think through applications of neural networks within your own team, I hope you find both uses for them in both structured and unstructured data.

Neural networks have transformed supervised learning and are creating tremendous economic value. It turns out though that the basic technical ideas behind neural networks have mostly been around for many decades. So why is it that they're just now taking off and working so well? In the next class, we'll talk about why it's only quite recently that neural networks have become this incredibly powerful tool that you can use.

<end>