这个函数的主要目的是对
p
a
r
a
m
e
t
e
r
s
\ parameters
parameters里的所有参数的梯度进行规范化;
p
a
r
a
m
e
t
e
r
s
\ parameters
parameters为要规范的参数,
m
a
x
_
n
o
r
m
\ max\_norm
max_norm为所有参数的梯度的范数的上界,
n
o
r
m
_
t
y
p
e
\ norm\_type
norm_type为范数的类型,当
n
o
r
m
_
t
y
p
e
=
i
n
f
\ norm\_type=inf
norm_type=inf时,代表无穷范数;
设
p
a
r
a
m
e
t
e
r
s
\ parameters
parameters里所有参数的梯度的范数为
t
o
t
a
l
_
n
o
r
m
\ total\_norm
total_norm, 若
m
a
x
_
n
o
r
m
>
t
o
t
a
l
_
n
o
r
m
,
\ max\_norm > total\_norm,
max_norm>total_norm,
p
a
r
a
m
e
t
e
r
s
\ parameters
parameters里面的参数的梯度不做改变; 若
m
a
x
_
n
o
r
m
<
t
o
t
a
l
_
n
o
r
m
,
\ max\_norm < total\_norm,
max_norm<total_norm,
p
a
r
a
m
e
t
e
r
s
\ parameters
parameters里面的参数的梯度都要乘以一个系数
c
l
i
p
_
c
o
e
f
\ clip\_coef
clip_coef,
c
l
i
p
_
c
o
e
f
=
m
a
x
_
n
o
r
m
/
(
t
o
t
a
l
_
n
o
r
m
+
1
e
−
6
)
\ clip\_coef=max\_norm / (total\_norm + 1e-6)
clip_coef=max_norm/(total_norm+1e−6)
附上源码:(注意源码中的这句话"The norm is computed over all gradients together"——计算所有梯度的范数)
defclip_grad_norm_(parameters, max_norm, norm_type=2):r"""Clips gradient norm of an iterable of parameters.
The norm is computed over all gradients together, as if they were
concatenated into a single vector. Gradients are modified in-place.
Arguments:
parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a
single Tensor that will have gradients normalized
max_norm (float or int): max norm of the gradients
norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for
infinity norm.
Returns:
Total norm of the parameters (viewed as a single vector).
"""ifisinstance(parameters, torch.Tensor):
parameters =[parameters]#第一步
parameters =list(filter(lambda p: p.grad isnotNone, parameters))
max_norm =float(max_norm)
norm_type =float(norm_type)if norm_type == inf:
total_norm =max(p.grad.data.abs().max()for p in parameters)else:
total_norm =0for p in parameters:#第二步
param_norm = p.grad.data.norm(norm_type)#第三步
total_norm += param_norm.item()** norm_type
total_norm = total_norm **(1./ norm_type)
clip_coef = max_norm /(total_norm +1e-6)if clip_coef <1:for p in parameters:
p.grad.data.mul_(clip_coef)return total_norm