def sigmoid_rampup(current_epoch):
current = np.clip(current_epoch, 0.0, 15.0)
phase = 1.0 - current / 15.0
return np.exp(-5.0 * phase * phase).astype(np.float32)
if __name__ == '__main__':
x = np.linspace(1,20, num=20)
y = sigmoid_rampup(x)
plt.plot(x,y)
plt.show()
在SE-SSD 的代码中看到了 sigmoid_rampup 函数, 它用来调节 consistency_weight 在整个loss中的权重.
这个思想来自于 Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, pages
1195–1204, 2017. 2, 6
Training details We adopt the ADAM optimizer and cosine annealing learning rate [18] with a batch size of four for 60 epochs. We follow [27] to ramp up µt (Eq. (5)) from 0 to 1 in the first 15 epoches using a sigmoid-shaped function e−5(1−x)2.
consistency_weight = 1.0 * self.sigmoid_rampup(self.epoch)