1. ParameterCollection()
DyNet
最基础的概念之一是参数集合,其定义为如下:
class dynet.ParameterCollection(parent=None)
模型所需要参数的创建、加载和保存都通过此类对象实现,是普通参数和查找表参数的容器。dy.Trainer
对象认为ParameterCollection
对象中的参数是需要通过训练学习得到的。
add_lookup_parameters(dim=(vocab_szie,emb_size), init=None, name='')
向ParameterCollection
对象添加查找表参数,默认初始化方式为GlorotInitializer
。下面的函数功能与之类似,函数返回值类型都是dynet.Parameters
对象:
add_parameters(dim, init=None, name='')
1.1 paras.as_array()
通过以上两种方式创建的参数类型均为dynet.Parameters
对象,具有函数as_array()
,返回值为参数对应的ndarray
类型
2. Expressions
Expressions are the building block of a Dynet computation graph.
Expressions are the main data types being manipulated in a DyNet program. Each expression represents a sub-computation in a computation graph.
Expressions are used as an interface to the various functions that can be used to build DyNet computation graphs.
2.1 dynet.parameter()
dynet.parameter(*args)
Add parameters to the computation graph.
Get the expression objects corresponding to parameters. Gradients for parameters will be computed and used by Optimizers to update.
因为通过ParameterCollection的对象并没有加入到计算图中,因此需要上述函数将参数添加进computation graph。函数返回值类型是expression
2.2 dynet.concatenate()
参数xs
为需要串接的向量或矩阵元素列表,在函数执行过程中会将需要使用的向量元素加入到计算图中。在nlp应用中因此就不需要专门将dynet.LookupParameters
添加进计算图中,而相较之下普通的模型参数dynet.Parameters
就需要专门函数语句来将其加入计算图中。当需要串接矩阵时(通常是在minibatch情况下),串接操作作用在矩阵的dim=0维度上,即将矩阵按列进行串接起来,输入矩阵的维度应为 [# of vectors to be concatenated, #batches]。
3. Operations
Operations are used to build expressions.
3.1 dynet.affine_transform()
计算仿射变换,输入参数为列表形式,且包含奇数个元素
- Parameters:exprs (
list
) – A list containing an odd number of expressions - Returns:An expression equal to: x s [ 0 ] + x s [ 1 ] ∗ x s [ 2 ] + . . . xs[0] + xs[1]*xs[2] + ... xs[0]+xs[1]∗xs[2]+...
- Return type:
dynet.Expression
3.2 dynet.pickneglogsoftmax()
Negative softmax log likelihood
This function takes in a vector of scores x x x, and performs a log softmax, takes the negative, and selects the likelihood corresponding to the element v v v. This is perhaps the most standard loss function for training neural networks to predict one out of a set of elements.
- Parameters:
-
x
x
x(
dynet.Expression
)- input scores -
v
v
v(
int
)- true class
-
x
x
x(
- Returns: − log e x v ∑ j e x j -\log\frac{e^{x_v}}{\sum_{j}e^{x_j}} −log∑jexjexv
- Return type:
dynet.Expression
3.3 dynet.log_softmax()
函数功能为输入向量的 subset 计算softmax;具体来讲就是没有包含在参数
r
e
s
t
r
i
c
t
restrict
restrict(形式为list
)中的元素将会被置为负无穷,包含在参数内的输入向量的子向量会被计算softmax,默认为输入向量的所有元素计算softmax。
- Parameters:
-
x
x
x (
dynet.Expression
) – Input expression -
r
e
s
t
r
i
c
t
restrict
restrict (
list
) – List of log softmax to compute (default: (None))
-
x
x
x (
- Returns: A vector with the log softmax over the specified elements
- Return type:
dynet.Expression
4. Optimizers
优化方法包括梯度下降系列的SGD、Adam、RMSProp等等,也包括dropout这样的tricks。
4.1 dynet.dropout()
Dropout
With a fixed probability, drop out (set to zero) nodes in the input expression, and scale the remaining nodes by 1 p \frac{1}{p} p1.
- Parameters:
-
x
x
x (
dynet.Expression
) – Input expression - p p p (number) – The dropout probability
-
x
x
x (
- Returns:The dropped out expression
- Return type:
dynet.Expression