DARTS:DIFFERENTIABLE ARCHITECTURE SEARCH

1. Introduction

before: discovering state-of-the-art neural network architecture requires substantial effort of human expers.
Contribution:
1.We introduce a novel algorithm for differentable network architecture search based on bilevel optimization(二阶优化) .propose a method for efficient architecture search called DARTs and relax the search space to be continuous,so tha the atchitecture can be optimized;
2.We achieve remarkable efficiency impovement.Through extensive experiments on image classification and language modeling tasks we show that gradient-based architecture search achieve highly competitive results on CIFAR-10 and outperforms the state of the art on PTB;
The code of DARTS is available to us at https://github.com/quark0/darts.

https://github.com/quark0/darts

2.Diffrerentiable architecture search

2.1.SEARCH SPACE

We search for a computation cell as the building block of the final architecture. The learned cell could either be stacked to form a convolutional network or connected to form a recurrent network.
cell: a cell is a directed acyclic （无循环）graph consisting of an ordered sequence of N nodes;
node:$x^{(i)}$ is a latent representation(e.g. a feature map in convolution network);
directed edge:$(i,j)$;
operation:$o^{(i,j)}$ is a operation that transform $x^{(i)}$.
Construction:
1.the cell has two input nodes and a single output node;
2.the input nodes are defined as the cell outputs in the previous two layer;
Each intermediate node is compued based on all of its predecessors:
$$\begin{array}{c}{x}^{(j)}=\sum _{i<j}{o}^{(i,j)}(x^{(i)})\end{array}$$

2.2. Continuous Relaxation and Optimization

2.2.1 meaning

1. O:a set of candidate operation
2. $o(\cdot )$is some function which is applied to $x^{(i)}$

2.2.2 the function is below:

$$\begin{array}{c}{\bar{o}}^{(i,j)}(x)=\sum _{o\in O}\frac{exp({\alpha }_0^{(i,j)})}{{\sum }_{o^{{}^{\prime }}\in O}exp({\alpha }_{o_{{}^{\prime }}}^{(i,j)})}\end{array}$$
1.$(i,j)$are parameterized by a vector ${\alpha }^{(i,j)}$ of dimension |O|.
2.The task of architecture search reduces to learning a set of continuous variables α = { α ( i , j ) } \alpha=\{\alpha^{(i,j)}\} α={α(i,j)}
3.at the end of search, a discrete architecture can be obtained by replacing each mixed operation ${\bar{o}}^{i,j}$, at the most of time ,$o^{(i,j)}=argma{x}_{o\in O}{\alpha }_o^{(i,j)}$
After relaxation, our goal is to joinly learn the architecture $\alpha$ and the weight $w$.
$L_{train}$:training loss;
$L_{val}$:validation loss;
Both losses are determined not only by the architecture $\alpha$,but also the weights w in the network.

2.2.3 Goals

1. ${\alpha }^{\ast }$--------------------minimize the validation loss $L_{val}(w^{\ast },{\alpha }^{\ast })$
2. $w^{\ast }$--------------------minimize the training loss $L_{train}(w,{\alpha }^{\ast })$

It is a bilevel optimization problem with $\alpha$ as the upper-level variable and $w$ as the lower-level variable:
$$\begin{array}{c}\begin{array}{rl}& \underset{\alpha }{\min }{L}_{val}(w^{\ast }(\alpha ),\alpha )\\ & s.t.w^{\ast }(\alpha )=argmi{n}_w{L}_{train}(w,\alpha )\end{array}\end{array}$$
Algorithm:

2.3 Approximate Architecture Gradient

Evaluating the architecture gradient exactly can be prohibitive due to the expensive inner optimizaton.So we propose a simple approximate scheme as follows:
$$\begin{array}{rl}& {\nabla }_{\alpha }{L}_{val}(w^{\ast }(\alpha ),\alpha )\\ \approx & {\nabla }_{\alpha }{L}_{val}(w-\xi {\nabla }_w{L}_{tran}(w,\alpha ),\alpha )\end{array}$$
meaning:

1. $w$:current weights maintained by the algorithm；
2. $\xi$:the learning rate for a step of inner optimization；
3. ${\nabla }_{\alpha }{L}_{val}(w,\alpha )$：if $w$ is already a local optimum（局部最优） for the inner optimization,and thus ${\nabla }_w{L}_{train}(w,\alpha )$=0,so $w=w^{\ast }(\alpha )$ and $w^{\ast }(\alpha )$=$w-\xi {\nabla }_w{L}_{train}(w,\alpha )$；

Idea
approximate $w^{\ast }$ by adapting $w$ using only a single training step, without solving the inner optimization completely bu training until convergence.
while we are not currently aware of the convergence guarantees for out optimization algorithm, in practice it is able to reach a fixed point with a suitable choixe of $\xi$.
citation: https://zhuanlan.zhihu.com/p/156832334

2.3.1 prodcure

1. Firstly,${\nabla }_{\alpha }{L}_{val}(w-\xi {\nabla }_w{L}_{train}(w,\alpha ).\alpha )$ can be simplized as ${\nabla }_{\alpha }f(g_1(\alpha ),g_2(\alpha ))$
   $$\begin{gathered} \cdot f(\cdot ,\cdot )=L_{val}(\cdot ,\cdot ) \\ \cdot g_1(\alpha )=w-\xi L_{train}(w,\alpha ) \\ \cdot g_2(\alpha )=\alpha \end{gathered}$$

$$\begin{gathered} {\nabla }_{\alpha }f(g_1(\alpha ),g_2(\alpha ))={\nabla }_{\alpha }{g}_1(\alpha )\cdot D_{1}f(g_1(\alpha ),g_2({\alpha }_2))+{\nabla }_{\alpha }{g}_2(\alpha )\cdot D_{2}f(g_1(\alpha ),g_2({\alpha }_2)) \\ {\nabla }_{\alpha }{g}_1(\alpha )=-\xi {\nabla }_{\alpha ,w2}^2{L}_{train}(w,\alpha ) \\ {\nabla }_{\alpha }{g}_2(\alpha )=1 \\ {D}_{1}f(g_1(\alpha ),g_2(\alpha ))={\nabla }_{w^{,}}{L}_{val}(w^{,},\alpha ) \\ {D}_{2}f(g_1(\alpha ),g_2(\alpha ))={\nabla }_{\alpha }{L}_{val}(w^{,},\alpha ) \\ {\nabla }_{\alpha }{L}_{val}(w-\xi {\nabla }_w{L}_{train}(w,\alpha ),\alpha )={\nabla }_{\alpha }{L}_{val}(w^{\prime },\alpha )-\xi {\nabla }_{\alpha ,w}^2\cdot {\nabla }_{w^{\prime }}{L}_{val}(w^{\prime },\alpha ) \end{gathered}$$
3.
$$\begin{gathered} s.t.ϵ=0.01/\Vert {\nabla }_{w^{\prime },\alpha }{\Vert }_2 \\ {\nabla }_{\alpha ,w}^2{L}_{train}(w,\alpha )\cdot {\nabla }_{w^{\prime }}{L}_{val}(w^{\prime },\alpha )\approx \frac{{\nabla }_{\alpha }{L}_{train}(w^{+},\alpha )-{\nabla }_{\alpha }{L}_{train}(w^{-},\alpha )}{2ϵ} \\ {w}^{-/+}=w-/+ϵ{\nabla }_{w^{\prime }}{L}_{val}(w^{\prime },\alpha ) \\ {f}^{\prime }(x_0)\cdot A\approx \frac{f(x_0+hA)-f(x_0-hA)}{2h} \end{gathered}$$