PyTorch - Python deep learning neural network API

Welcome back to this series on neural network programming with PyTorch. To kick this series off, let’s introduce PyTorch, a deep learning neural network package for Python. There’s no better place to start as we’ll be using PyTorch in this series to program our neural networks. Without further ado, let’s get started.

PyTorch is a deep learning framework and a scientific computing package. This is how the PyTorch core team describes PyTorch, anyway. The scientific computing aspect of PyTorch is primarily a result PyTorch’s tensor library and associated tensor operations.

A tensor is an n-dimensional array

For example, PyTorch torch.Tensor objects that are created from NumPy ndarray objects, share memory. This makes the transition between PyTorch and NumPy very cheap from a performance perspective.

With PyTorch tensors, GPU support is built-in. It’s very easy with PyTorch to move tensors to and from a GPU if we have one installed on our system.

PyTorch tensor operations can be performed on a GPU

We’ll talk more about GPUs and why we use them in deep learning in the post on CUDA. For now, just know that PyTorch tensors and their associated operations are very similar to NumPy n-dimensional arrays.

Tensors are super important for deep learning and neural networks because they are the data structure that we ultimately use for building and training our neural networks.

On top of the tensor library, PyTorch has much more to offer in terms of building and training neural networks. Before we touch on the deep learning specifics of PyTorch, let’s look at some details on how PyTorch was created.

PyTorch: A brief history

The initial release of PyTorch was in October of 2016, and before PyTorch was created, there was and still is, another framework called Torch. Torch is a machine learning framework that’s been around for quite a while and is based on the Lua programming language.

The connection between PyTorch and this Lua version, called Torch, exists because many of the developers who maintain the Lua version are the individuals who created PyTorch.

Soumith Chintala is credited with bootstrapping the PyTorch project, and his reason for creating PyTorch is pretty simple, the Lua version of Torch was aging, and so a newer version written in Python was needed. As a result, PyTorch came to be.

Facebook created PyTorch

One thing that you may hear about PyTorch is that it was created and is maintained by Facebook. This is because Soumith Chintala worked at Facebook AI Research when PyTorch was created (still does at the time of this writing). However, there are many other companies with a vested interest in PyTorch.

The PyTorch GitHub repo indicates that there are quite a few contributors, upwards of seven hundred at the current moment. Right near the top of the contributions by commit, we have Soumith, but there are many others.

Let’s look now at the deep learning features of PyTorch.

Deep learning with PyTorch

This table gives us a list of PyTorch packages and their corresponding descriptions. These are the primary PyTorch components we’ll be learning about and using as we build neural networks in this series.

This table gives us a list of PyTorch packages and their corresponding descriptions. These are the primary PyTorch components we’ll be learning about and using as we build neural networks in this series.

Package	Description
torch	The top-level PyTorch package and tensor library
torch.nn	A subpackage that contains modules and extensible classes for building neural networks
torch.autograd	A subpackage that supports all the differentiable Tensor operations in PyTorch
torch.nn.functional	A functional interface that contains typical operations used for building neural networks like loss functions, activation functions, and convolution operations
torch.optim	A subpackage that contains standard optimization operations like SGD and Adam
torch.utils	A subpackage that contains utility classes like data sets and data loaders that make data preprocessing easier
torchvision	A package that provides access to popular datasets, model architectures, and image transformations for computer vision

At the moment, the torchvision package is separate from the top-level torch package. However, this may change in the future if torchvision is pulled in as a subpackage of torch.

Why use PyTorch for deep learning?

Let’s talk about the prospects for learning PyTorch. For beginners to deep learning and neural networks, the top reason for learning PyTorch is that it is a thin framework that stays out of the way.

PyTorch is thin and stays out of the way!

When we build neural networks with PyTorch, we are super close to programming neural networks from scratch. The experience of programming in PyTorch is as close as it gets to the real thing.

PyTorch is as close as it gets to the real thing!

After understanding the process of programming neural networks with PyTorch, it’s pretty easy to see how the process works from scratch in say pure Python. This is why PyTorch is great for beginners.

After using PyTorch, you’ll have a much deeper understanding of neural networks and the deep learning. One of the top philosophies of PyTorch is to stay out of the way, and this makes it so that we can focus on neural networks and less on the actual framework.

Philosophy of PyTorch

As of the writing of this post, PyTorch’s development is guided by the following list:

• Stay out of the way
• Cater to the impatient
• Promote linear code-flow
• Full interop with the Python ecosystem
• Be as fast as anything else .

The fact that PyTorch stays out of the way makes PyTorch well suited for deepening our understanding of neural networks. When we write PyTorch code, we are just writing and extending standard Python classes, and when we debug PyTorch code, we are using the standard Python debugger.

PyTorch’s design is modern, Pythonic, and thin. The source code is easy to read for Python developers because it’s written mostly in Python, and only drops into C++ and CUDA code for operations that are performance bottlenecks.

Overall, PyTorch is a great tool for deepening our understanding of deep learning and neural networks.

Investing in PyTorch as a deep learning framework

From a knowledge investment perspective, PyTorch can be seen as a safer option simply because Facebook is backing it, and it’s built for Python, which unlike Lua, has a large and growing deep learning community.

In addition to the Facebook and Python edge, PyTorch is super thin and highly integrated with Python and very thin, which makes it more likely that PyTorch will be capable of adapting to the rapidly evolving deep learning environment as things change over time.

These characteristics promote the longevity of PyTorch as a deep learning framework.

PyTorch for deep learning research

A common PyTorch characteristic that often pops up is that it’s great for research. The reason for this research suitability has do do with a technical design consideration. To optimize neural networks, we need to calculate derivatives, and to do this computationally, deep learning frameworks use what are called computational graphs.

Computational graphs are used to graph the function operations that occur on tensors inside neural networks.

These graphs are then used to compute the derivatives needed to optimize the neural network. PyTorch uses a computational graph that is called a dynamic computational graph. This means that the graph is generated on the fly as the operations are created.

This is in contrast to static graphs that are fully determined before the actual operations occur.

It just so happens that many of the cutting edge research topics in deep learning are requiring or benefiting greatly from dynamic graphs.