LR with spark liblinear

Logistic Regression

sparkliblinear 库的类关系图

1、LR

Given a set of training label-instance pairs ${(x_ i ,y_ i )}^ l_{i=1} ,
x i \in \mathbb{R}^ n , y i \in{−1,1}, \forall{i} $

LR with L2 reg model considers the following optimization problem:

minwf(w)=12wTw+Cl∑i=1log(1+exp(−yiwTxi))（1）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1） min w f ( w ) = 1 2 w T w + C ∑ i = 1 l l o g ( 1 + e x p ( − y i w T x i ) ) （ 1 ）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1）

minwf(w)=12wTw+C∑i=1llog(1+exp(−yiwTxi))（1）

$$\min_{w}f(w)=\frac{1}{2}{w}^T{w} + C\sum^l_{i=1}log(1+exp(-y_i{w}^T{x}_i)) \qquad （1）$$

2、A Trust Region Newton Method(TRON)

TRON obtains the truncated Newton step by approximately solving
mindqt(d)subject to‖d‖≤Δt（2）

mindqt(d)subject to∥d∥≤Δt（2） min d q t ( d ) subject to ‖ d ‖ ≤ Δ t （ 2 ）

mindqt(d)subject to∥d∥≤Δt（2）

mindqt(d)subject to∥d∥≤Δt（2）

mindqt(d)subject to∥d∥≤Δt（2）

mindqt(d)subject to∥d∥≤Δt（2）

mindqt(d)subject to∥d∥≤Δt（2）

mindqt(d)subject to∥d∥≤Δt（2）

mindqt(d)subject to∥d∥≤Δt（2）

mindqt(d)subject to∥d∥≤Δt（2）

mindqt(d)subject to∥d∥≤Δt（2）

$$\min_d q_t(d)\qquad \text{subject to}\qquad \|d\|\le\Delta_t \qquad （2）$$

Δt Δt Δ t Δt Δt Δt Δt Δt Δt Δt Δt Δt $\Delta_t$ is the size of the trust region, qt(d)=∇f(wt)Td+12dT∇2f(xt)d qt(d)=∇f(wt)Td+12dT∇2f(xt)d q t ( d ) = ∇ f ( w t ) T d + 1 2 d T ∇ 2 f ( x t ) d qt(d)=∇f(wt)Td+12dT∇2f(xt)d qt(d)=∇f(wt)Td+12dT∇2f(xt)d qt(d)=∇f(wt)Td+12dT∇2f(xt)d qt(d)=∇f(wt)Td+12dT∇2f(xt)d qt(d)=∇f(wt)Td+12dT∇2f(xt)d qt(d)=∇f(wt)Td+12dT∇2f(xt)d qt(d)=∇f(wt)Td+12dT∇2f(xt)d qt(d)=∇f(wt)Td+12dT∇2f(xt)d qt(d)=∇f(wt)Td+12dT∇2f(xt)d $q_t(d) = \nabla f(w^t)^Td+\frac{1}{2}d^T\nabla^2f(x^t)d$
is the second-order Taylor approximation of f(wt+d)−f(wt) f(wt+d)−f(wt) f ( w t + d ) − f ( w t ) f(wt+d)−f(wt) f(wt+d)−f(wt) f(wt+d)−f(wt) f(wt+d)−f(wt) f(wt+d)−f(wt) f(wt+d)−f(wt) f(wt+d)−f(wt) f(wt+d)−f(wt) f(wt+d)−f(wt) $f(w_t + d) − f(w_t )$.

applying CG(Conjugate Gradient) to slove (2)

2.1 Distributed Algorithm

We partition the data matrix X and the labels Y into
disjoint p parts.
X=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]T

X = [ X 1 , . . . , X p ] T , Y = d i a g ( y 1 , . . . , y l ) = [ Y 1... Y p ] , σ ( v ) ≡ [ 1 + e x p ( − v 1 ) , . . . , 1 + e x p ( − v n ) ] T

X=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]TX=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]T

X=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]TX=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]T

X=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]TX=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]T

X=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]TX=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]T

X=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]TX=[X1,...,Xp]T,Y=diag(y1,...,yl)=[Y1...Yp],σ(v)≡[1+exp(−v1),...,1+exp(−vn)]T

$$X = [X_1 ,...,X_p ]^T ,\\ Y = diag(y_1 ,...,y_l ) =[Y 1...Y p],\\ σ(v) ≡ [1 + exp(−v_1 ),...,1 + exp(−v_n )]^T$$

We can observe that for computing (12)-(14), only the data partition Xk Xk X k Xk Xk Xk Xk Xk Xk Xk Xk Xk $X_k$ is needed in computing. Therefore, the computation can be done in parallel, with the partitions being stored distributedly. After the map functions are computed, we need to reduce the results to the machine performing the TRON algorithm in order to obtain the summation over all partitions.

3、Implement Design

1) Loop Structure: choose the while loop to implement the software

2) Data Encapsulation:

AA uses two arrays to store indices and feature values of an instance:

ndex1 index2 index3 index4 index5 …

value1 value2 value3 value4 value5 …

3) Using mapPartitions Rather Than map

4) not to cache σ(YkXkw) σ(YkXkw) σ ( Y k X k w ) σ(YkXkw) σ(YkXkw) σ(YkXkw) σ(YkXkw) σ(YkXkw) σ(YkXkw) σ(YkXkw) σ(YkXkw) σ(YkXkw) $σ(Y_kX_kw)$

5) Using Broadcast Variables