Pytorch学习

Pytorch学习

一、 tensor

tensor是pytorch最基本的数据结构，是pytorch中一个类（torch.Tensor），很多属性与方法和numpy中的数据相似。因此在numpy基础学习pytorch容易上手。
Tensors are a specialized data structure that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters.

Tensors are similar to NumPy’s ndarrays, except that tensors can run on GPUs or other hardware accelerators. In fact, tensors and NumPy arrays can often share the same underlying memory, eliminating the need to copy data (see Bridge with NumPy). Tensors are also optimized for automatic differentiation (we’ll see more about that later in the Autograd section). If you’re familiar with ndarrays, you’ll be right at home with the Tensor API. If not, follow along!

Tensors是特殊的数据结构，相似于数组（array)和矩阵（matrices）。在pytorch，使用tensor编码模型的输入、输出及模型参数；
Tensors相似与numpy的数组（ndarray），除了tensors能运行在GPU或其它硬件加速器上。实际上tensors和numpy array 能够经常共享相同的底层内存，这样消除了数据复制的需求。tensor也可用自动微分的优化。如果你对numpy ndarray熟悉，那么你蒋对Tensor API非常熟悉。

创建Tensors

Tensors can be initialized in various ways. Take a look at the following examples
tensor能通过多种方式创建，看看下面几个例子

1. directly from data

Tensors can be created directly from data. The data type is automatically inferred.
tensor通过列表直接创建，其类型自动获得
例如1.1
import torch
torch.tensor([1,2])

2. from numpy array

Tensors can be created from a numpy ndarray
例1.2
import torch
import numpy as np
data = np.array([1,2])
torch.tensor(data)

3. from other tensors

The new tensor retains the properties (shape, datatype) of the argument tensor, unless explicitly overridden.
例1.3
import torch
torch.ones_like(data)
torch.rand_like(data, dtype=torch.float)

4. with constant or random value

shape is a tuple of tensor dimensions. In the functions below, it determines the dimensionality of the output tensor.
例1.4
import torch
shape = (2,3,)
torch.ones(shape)
torch.zeros(shape)
torch.rand(shape)

Tensor的属性

Tensor attributes describe their shape, datatype, and the device on which they are stored.
有哪些属性？查看属性？改变或设置属性

有哪些属性

1. dtype:

dtype是torch一个类，torch提供了12种数据类型类: torch.float32 or torch.float(默认的)，torch.float64 or torch.double, torch. int32, torch.bool, 每一个类型有相应的构造方法，如torch.Float32Tensor or torch.cuda.FloatTensor, torch.BoolTensor ort torch.cuda.BoolTensor， 有cuda表示在GPU上创建，没有cuda表示在cpu上创建。

2. shape

3. device

device是torch中一个类，说明Tensor存储的位置。Tensor默认存在cpu中。创建device对象
device也是torch的一个类，A torch.device is an object representing the device on which a torch.Tensor is or will be allocated.
a. 创建device对象
torch.device(“cuda”)
torch.device(“cuda:0”)
torch.device(“cpu”)
torch.device(1) # 表示在第2个GPU, ordinal只适合GPU
torch.device(“cuda”,0)
b. Tensors可以在定义时指明设备，默认是CPU
torch.tensor([1,2],device=“cuda:0”)
torch.tensor([1,2], device=torch.device(“cuda:0”))
也可以移动到指定设备
torch.tensor([1,2]).to(“cuda:0”)
torch.tensor([1,2]).cpu()
torch.tensor([1,2]).cuda()
If the device ordinal is not present, this object will always represent the current device for the device type, even after torch.cuda.set_device() is called; e.g., a torch.Tensor constructed with device ‘cuda’ is equivalent to ‘cuda:X’ where X is the result of torch.cuda.current_device().

4. requires_grad 和 grad_fn 和grad属性

每个Tensor在创建时可以指明requires_grad的属性，如为True，表示可以求导。默认为False。
也可以通过Tensor.requires_grad=True 改变其值
例
data = torch.randn(2,3, requires_grad=True)
data.requires_grad = False
由requires_grad为True的Tensor构建的函数具有grad_fn属性，grad_fn指向这个函数。
函数对象调用backward()后，Tensor的grad属性才不为none

5. .T, mT, mH

Tensor.T Returns a view of this tensor with its dimensions reversed.
If n is the number of dimensions in x, x.T is equivalent to x.permute(n-1, n-2, …, 0).

Tensor.H Returns a view of a matrix (2-D tensor) conjugated and transposed.
x.H is equivalent to x.transpose(0, 1).conj() for complex matrices and x.transpose(0, 1) for real matrices.

Tensor.mT Returns a view of this tensor with the last two dimensions transposed.
x.mT is equivalent to x.transpose(-2, -1).

Tensor.mH
Accessing this property is equivalent to calling adjoint().

Tensor 方法

Over 100 tensor operations, including arithmetic, linear algebra, matrix manipulation (transposing, indexing, slicing), sampling and more are comprehensively described here.
有100以上Tensor操作，包括算术的，线性代数，矩阵，抽样和更多的在这里介绍

By default, tensors are created on the CPU. We need to explicitly move tensors to the GPU using .to method (after checking for GPU availability). Keep in mind that copying large tensors across devices can be expensive in terms of time and memory!
默认，Tensor创建在cpu上。也也可以通过.to() 方法显示地移动Tensor到GPU上，但是必须是在检查GPU可得性后。记住在不同设备间复制大的Tensor在时间和内存上 开销很大。
例
if torch.cuda.is_available():
tensor = tensor.to(“cuda”)

Try out some of the operations from the list. If you’re familiar with the NumPy API, you’ll find the Tensor API a breeze to use.

1 算术运算

+，-，*，**，/，sum,abs, ceil, floor, max, min, argmin, argmax, 三角函数， 反三角函数，指数函数，对数函数，统计运算

2 关系运算

<==, all, any , where,

3 位运算

4 逻辑运算

3 代数运算

4 形状运算

torch.tensor_split()， Tensor.chunk(), sequeeze, unsequeeze, transpose, permute， Tensor.tile， Tensor.view，Tensor.reshape, Tensor.resize

4.1. Tensor.reshape()，view(), reshape_as(), view_as()

Returns a tensor with the same data and number of elements as self but with the specified shape. This method returns a view if shape is compatible with the current shape.
例
reshape 可以产生Tensor的视图或副本。若Tensor 是contiguous，则返回一个视图，否则返回一个副本
view产生一个视图，但前提条件是Tensor必须是contiguous的，否则产生错误

A single dimension may be -1, in which case it’s inferred from the remaining dimensions and the number of elements in input.

Basic slicing and indexing op, e.g. tensor[0, 2:, 1:7:2] returns a view of base tensor, see note below.
adjoint(), as_strided(), detach(), diagonal(), expand(), expand_as(),movedim(), narrow(), permute(), select(), squeeze(), transpose(), t()
T, H, mT, mH, real, imag
view_as_real()
unfold()
unsqueeze()
unbind()
split()
hsplit()
vsplit()
tensor_split()
split_with_sizes()
swapaxes()
swapdims()
chunk()
indices() (sparse tensor only)
values() (sparse tensor only)

4.2 Tensor.ravel，unsqueeze()，squeeze()，expand(), repeat(), tile()

Tensor.ravel:
Tensor.unfold: Returns a view of the original tensor which contains all slices of size size from self tensor in the dimension dimension
例
x = torch.arange(1., 8)
x.unfold(0, 2, 1)， # 按照在第0dimension，2个数字，间隔1
例
x = torch.randn(2,5)
x.unfold(1,2,2)
Tensor.expand(): Returns a new view of the self tensor with singleton dimensions expanded to a larger size.-1 表示相应维度不变。只能对为1的维度进行操作。
Tensor.repeat(): Repeats this tensor along the specified dimensions.1 表示该维度不变，类似广播机制
Tensor.tile(): Constructs a tensor by repeating the elements of input. The dims argument specifies the number of repetitions in each dimension.If dims specifies fewer dimensions than input has, then ones are prepended to dims until all dimensions are specified. For example, if input has shape (8, 6, 4, 2) and dims is (2, 2), then dims is treated as (1, 1, 2, 2).Analogously, if input has fewer dimensions than dims specifies, then input is treated as if it were unsqueezed at dimension zero until it has as many dimensions as dims specifies

4.3 t(), transpose(), swapaxes(), swapdims()， movedim, moveaxis, narrow(), permute()

transpose(input, dim0, dim1): Returns a tensor that is a transposed version of input. The given dimensions dim0 and dim1 are swapped.
swapaxes(): This function is equivalent to NumPy’s swapaxes function.
swapdims(input, dim0, dim1):
movedim(input, source, destination): Moves the dimension(s) of input at the position(s) in source to the position(s) in destination.
moveaxis(input, source, destination):
例
torch.movedim(t, (1, 2), (0, 1))
Tensor.narrow(dimension, start, length):
例
``

x = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
x.narrow(0, 0, 2)
tensor([[ 1, 2, 3],
[ 4, 5, 6]])
x.narrow(1, 1, 2)
tensor([[ 2, 3],
[ 5, 6],
[ 8, 9]])

Tensor.permute(): Returns a view of the original tensor input with its dimensions permuted.
例
```cpp
>>> x = torch.randn(2, 3, 5)
>>> x.size()
torch.Size([2, 3, 5])
>>> torch.permute(x, (2, 0, 1)).size()
torch.Size([5, 2, 3])

4.4 Tensor分割、合并 split, torch.stack, torch.cat, torch.dstack，chunk，unbind

split(tensor, split_size_or_sections, dim=0): Splits the tensor into chunks. Each chunk is a view of the original tensor.
If split_size_or_sections is an integer type, then tensor will be split into equally sized chunks (if possible). Last chunk will be smaller if the tensor size along the given dimension dim is not divisible by split_size.

If split_size_or_sections is a list, then tensor will be split into len(split_size_or_sections) chunks with sizes in dim according to split_size_or_sections.
unbind()
hsplit()
vsplit()
tensor_split(): Splits a tensor into multiple sub-tensors, all of which are views of input, along dimension dim according to the indices or number of sections specified by indices_or_sections.

torch.stack
torch.cat, torch.concat: Concatenates the given sequence of seq tensors in the given dimension. All tensors must either have the same shape (except in the concatenating dimension) or be empty.
torch.dstack, torch.hstack: Concatenates a sequence of tensors along a new dimension.

torch.unbind(input, dim=0) ： Removes a tensor dimension. Returns a tuple of all slices along a given dimension, already without it.

5 梯度相关计算

detach，backward， Tensor.retain_grad，Tensor.retains_grad， torch.Tensor.is_leaf

detach

返回一个新的Variable，从当前计算图中分离下来的，但是仍指向原变量的存放位置,不同之处只是requires_grad为false，得到的这个Variable永远不需要计算其梯度，不具有grad。
两个Variable具有相同的data，修改会反应在两个Variables

import torch

a = torch.tensor([1, 2, 3.], requires_grad=True)
print(a.grad)
out = a.sigmoid()

out.sum().backward()
print(a.grad)
返回：

(deeplearning) userdeMBP:pytorch user$ python test.py 
None
tensor([0.1966, 0.1050, 0.0452])

当使用detach()但是没有进行更改时，并不会影响backward():

import torch

a = torch.tensor([1, 2, 3.], requires_grad=True)
print(a.grad)
out = a.sigmoid()
print(out)

#添加detach(),c的requires_grad为False
c = out.detach()
print(c)

#这时候没有对c进行更改，所以并不会影响backward()
out.sum().backward()
print(a.grad)
返回：

(deeplearning) userdeMBP:pytorch user$ python test.py 
None
tensor([0.7311, 0.8808, 0.9526], grad_fn=<SigmoidBackward>)
tensor([0.7311, 0.8808, 0.9526])
tensor([0.1966, 0.1050, 0.0452])

如果这里使用的是c进行sum()操作并进行backward()，则会报错：

import torch

a = torch.tensor([1, 2, 3.], requires_grad=True)
print(a.grad)
out = a.sigmoid()
print(out)

#添加detach(),c的requires_grad为False
c = out.detach()
print(c)

#使用新生成的Variable进行反向传播
c.sum().backward()
print(a.grad)
返回：

(deeplearning) userdeMBP:pytorch user$ python test.py 
None
tensor([0.7311, 0.8808, 0.9526], grad_fn=<SigmoidBackward>)
tensor([0.7311, 0.8808, 0.9526])
Traceback (most recent call last):
  File "test.py", line 13, in <module>

import torch

a = torch.tensor([1, 2, 3.], requires_grad=True)
print(a.grad)
out = a.sigmoid()
print(out)

#添加detach(),c的requires_grad为False
c = out.detach()
print(c)
c.zero_() #使用in place函数对其进行修改

#会发现c的修改同时会影响out的值
print(c)
print(out)

#这时候对c进行更改，所以会影响backward()，这时候就不能进行backward()，会报错
out.sum().backward()
print(a.grad)
返回：

(deeplearning) userdeMBP:pytorch user$ python test.py 
None
tensor([0.7311, 0.8808, 0.9526], grad_fn=<SigmoidBackward>)
tensor([0.7311, 0.8808, 0.9526])
tensor([0., 0., 0.])
tensor([0., 0., 0.], grad_fn=<SigmoidBackward>)
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    out.sum().backward()

torch.no_grad()

Context-manager that disabled gradient calculation.

x = torch.tensor([1.], requires_grad=True)

例

with torch.no_grad():
… y = x * 2
y.requires_grad
False
@torch.no_grad()
… def doubler(x):
… return x * 2
z = doubler(x)
z.requires_grad

torch.enable_grad

Context-manager that enables gradient calculation.

Enables gradient calculation, if it has been disabled via no_grad or set_grad_enabled.

>>> x = torch.tensor([1.], requires_grad=True)
>>> with torch.no_grad():
...   with torch.enable_grad():
...     y = x * 2
>>> y.requires_grad
True
>>> y.backward()
>>> x.grad
>>> @torch.enable_grad()
... def doubler(x):
...     return x * 2
>>> with torch.no_grad():
...     z = doubler(x)
>>> z.requires_grad
True

```python


##### torch.set_grad_enabled(mode)`
Context