一、卷积层
注意 使用Batch Normalization时不需要bias 每个输出通道一个数值作为bias
参数量
P = Cout*Cin*K*K+bias
bias = Cout
计算量
步幅为1
乘 加 偏置
FlOPs = [(Cin*k*k)+(Cin*k*k-1)+1]*Cout*W*H
记乘加为浮点运算
FLOPs = Cin*k*k*Cout*W*H
若步幅不为1 可以用输出图片宽高作为WH带入近似
二、全连接
参数量
P = In*Out+bias
bias = Out
计算量
乘 加 偏置
FLOPs = [In+(In-1)+1]*Out
记乘加为浮点运算
FLOPs = In*Out
共N次操作
FLOPs = N*In*Out
三、循环神经网络
注意双向要翻倍
参数量
x为输入维度
h为隐层维度
PRNN = (x+h)*h+bias
bias = h
PGRU = 3*PRNN
PLSTM = 4*PRNN
四、注意力
参数量
P = 4*(dim*Innerdim)
Innerdim = headdim*heads
计算量
[B,N,dim]->[B,N,Innerdim]
[B,N,Innerdim]->[B,heads,N,Innerdim/heads]
FLOPs = N*dim*Innerdim*3
Q*K'
FLOPs = heads*N*(Innerdim/heads)*N = N*N*Innerdim
Q*K'*V
FLOPs = heads*N*N*(Innerdim/heads) = N*N*Innerdim
FLOPs = N*dim*Innerdim
总共
FLOPs = 4*N*dim*Innerdim + 2*N*N*Innerdim = 2*N*Innerdim*(2*dim+N)