Tensors and Tensor Arithmetic

In deep learning it is common to see a lot of discussion around tensors as the cornerstone data structure. Tensor even appears in name of Google’s flagship machine learning library: TensorFlow. Tensors are a type of data structure used in linear algebra, and like vectors and matrices, you can calculate arithmetic operations with tensors. In this tutorial, you will discover what tensors are and how to manipulate them in Python with NumPy. After completing this tutorial, you will know:

• That tensors are a generalization of matrices and are represented using n-dimensional arrays.
• How to implement element-wise operations with tensors.
• How to perform the tensor product.

1.1 Tutorial Overview

This tutorial is divided into 3 parts ; they are:

1. What are Tensors

2. Tensors in Python

3. Tensor Arithmetic

4. Tensor Product

1.2 What are Tensors

A tensor is a generalization of vectors and matrices and is easily understood as a multidimensional array.

In the general case, an array of numbers arranged on a regular grid with a variable number of axes is known as a tensor.

A vector is a one-dimensional or first order tensor and a matrix is a two-dimensional or second order tensor. Tensor notation is much like matrix notation with a capital letter representing a tensor and lowercase letters with subscript integers representing scalar values within the tensor. For example, below defines a 3 × 3 × 3 three-dimensional tensor T with dimensions index as ti,j,k.

1.3 Tensors in Python

Like vectors and matrices, tensors can be represented in Python using the N-dimensional array (ndarray). A tensor can be defined in-line to the constructor of array() as a list of lists. The example below defines a 3 × 3 × 3 tensor as a NumPy ndarray. Three dimensions is easier to wrap your head around. Here, we first define rows, then a list of rows stacked as columns, then a list of columns stacked as levels in a cube.

# create tensor
from numpy import array
T = array([
    [[1, 2, 3], [4, 5, 6], [7, 8, 9]],
    [[11, 12, 13],[14, 15, 16], [17, 18, 19]],
    [[21, 22, 23], [24, 25, 26], [27, 28, 29]]
])
print(T.shape)
print(T)

Running the example first prints the shape of the tensor, then the values of the tensor itself. You can see that, at least in three-dimensions, the tensor is printed as a series of matrices, one for each layer. For this 3D tensor, axis 0 specifies the level (like height), axis 1 specifies the column, and axis 2 specifies the row.

1.4 Tensor Arithmetic

As with matrices, we can perform element-wise arithmetic between tensors. In this section,we will work through the four main arithmetic operations.

1.4.1 Tensor Addition

 

# Example of adding tensors
# tensor addition
from numpy import array
# define first tensor
A = array([
    [[1, 2, 3], [4, 5,6],[7,8,9]],
    [[11,12,13],[14,15,16],[17,18,19]],
    [[21, 22, 23],[24, 25, 26], [27,28, 29]]
])
# define second tensor
B = array([
    [[1, 2, 3], [4, 5,6],[7,8,9]],
    [[11,12,13],[14,15,16],[17,18,19]],
    [[21, 22, 23],[24, 25, 26], [27,28, 29]]
])

# add tensors
C = A + B
print(C)

Running the example prints the addition of the two parent tensors

1.4.2 Tensor Substraction

# Example of substracting tensors
# tensor substractions
from numpy import array
# define first tensor

# define first tensor
A = array([
    [[1, 2, 3], [4, 5,6],[7,8,9]],
    [[11,12,13],[14,15,16],[17,18,19]],
    [[21, 22, 23],[24, 25, 26], [27,28, 29]]
])
# define second tensor
B = array([
    [[1, 2, 3], [4, 5,6],[7,8,9]],
    [[11,12,13],[14,15,16],[17,18,19]],
    [[21, 22, 23],[24, 25, 26], [27,28, 29]]
])

# substract tensors
C = A - B
print(C)

Running the example prints the result of subtracting the first tensor from the second.

 1.4.3 Tensor Hadamard Product

# Example of Tensor Hadamard product
# tensor Hadamard product
from numpy import array

# define first tensor
A = array([
    [[1, 2, 3],  [4, 5,6], [7,8,9]],
    [[11,12,13], [14,15,16], [17,18,19]],
    [[21, 22, 23], [24, 25, 26],  [27,28, 29]]
])
# define second tensor
B = array([
    [[1, 2, 3],  [4, 5,6], [7,8,9]],
    [[11,12,13], [14,15,16], [17,18,19]],
    [[21, 22, 23], [24, 25, 26],  [27,28, 29]]
])

# mulyiply tensors
C = A * B
print(C)

 Running the example prints the result of multiplying the tensors.

1.4.4 Tensor Division

 

# Example of diving tensors
# tensor division
from numpy import array
# define first tensor
A = array([
    [[1, 2, 3],  [4, 5,6], [7,8,9]],
    [[11,12,13], [14,15,16], [17,18,19]],
    [[21, 22, 23], [24, 25, 26],  [27,28, 29]]
])
# define second tensor
B = array([
    [[1, 2, 3],  [4, 5,6], [7,8,9]],
    [[11,12,13], [14,15,16], [17,18,19]],
    [[21, 22, 23], [24, 25, 26],  [27,28, 29]]
])

# division tensors
C = A / B
print(C)

1.5 Tensor Product

 

 

# Example of tensor product
# tensor product
from numpy import array
from numpy import tensordot
# define first vector
A = array([1, 2])
# define second vector
B = array([3, 4])
# calculate tensor product
C = tensordot(A, B , axes=0)
print(C)

 Running the example prints the result of the tensor product. The result is an order-2 tensor (matrix) with the lengths 2 × 2.

 The tensor product is the most common form of tensor multiplication that you may encounter, but there are many other types of tensor multiplications that exist, such as the tensor dot product and the tensor contraction.

 1.6 Extensions

This section lists some ideas for extending the tutorial that you may wish to explore.

1.7 Summary

In this tutorial, you discovered what tensors are and how to manipulate them in Python with NumPy. Specifically, you learned:

• That tensors are a generalization of matrices and are represented using n-dimensional arrays.
• How to implement element-wise operations with tensors.
• How to perform the tensor product.