# 逻辑回归

## 原理

### 判断函数候选

y = 1 1 + e − z y = \frac{1}{1+e^{-z}}
z = w T x + b z = w^Tx+b

l ( w , b ) = ∑ i = 1 M l n p ( y i ∣ x i ; w , b ) l(w,b) = \sum_{i=1}^M ln p(y_i | x_i;w,b)

P w ( y = j ∣ x ) = e x p ( x T w ( j ) ) ∑ k = 1 K e x p ( x T w ( k ) ) P_w(y=j|x) = \frac{exp(x^Tw^{(j)})}{\sum_{k=1}^{K}exp(x^Tw^{(k)})}

### 损失函数

c o s t = − y l o g ( p ^ ) − ( 1 − y ) l o g ( 1 − p ^ ) cost = -ylog(\hat{p}) - (1-y)log(1-\hat{p}) .

w ∗ = a r g m a x x P ( Y ∣ X ) = a r g m a x w ∏ i = 1 M P ( Y i ∣ x i ) = a r g m a x w ∑ i = 1 M l o g P ( Y i ∣ x i ) = a r g m a x w ∑ i = 1 M [ y i l o g p 1 + ( 1 − y i ) l o g p 0 ] w^* = argmax_x P(Y|X) \\\\ = argmax_w \prod_{i=1}^{M} P(Y_i|x_i) \\\\ = argmax_w \sum_{i=1}^{M} log P(Y_i|x_i) \\\\ = argmax_w \sum_{i=1}^{M} [y_i log p_1 + (1-y_i) log p_0] \\\\

## 算法

### 逻辑回归算法

1. 随机初始化 θ \theta

2. 计算 θ j + 1 = θ j − α 1 m ∑ i = 1 m x i [ h ( x i ) − y i ] \theta_{j+1} = \theta_{j} - \alpha\frac{1}{m}\sum_{i=1}^{m}x_i[h(x_i)-y_i]

3. 迭代

### FTRL算法

1. 对于 i ∈ { i ∼ d } i\in\{i \sim d\} ，初始化 z i = 0 , n i = 0 z_i = 0, n_i = 0

2. 对样本t = 1 to T,

1. 计算 x t + 1 , i = { 0 , i f ∣ z t , i ∣ ≤ λ 1   − ( β + n i α + λ 2 ) − 1 ( z t , i − s i g n ( z t , i ) λ 1 ) , o t h e r w i s e x_{t+1,i} = \left\{\begin{aligned} &0 ,&if |z_{t,i}| \leq \lambda_1 \\\ & -(\frac{\beta+\sqrt{n_i}}{\alpha}+\lambda_2)^{-1}(z_{t,i} - sign(z_{t,i})\lambda_1),& otherwise\end{aligned}\right.

2. 计算 p t = σ ( x t w ) p_t = \sigma(x_tw) ，使用label函数和预测值 p t p_t 迭代

3. 对于i

g i = ( p t − y t ) x i g_i = (p_t - y_t)x_i

σ i = 1 α ( n i + g i 2 − n i ) \sigma_i = \frac{1}{\alpha}(\sqrt{n_i + g_i^2} - \sqrt{n_i})

z i = z i + g i − σ i w t , i z_i = z_i + g_i - \sigma_iw_{t,i}

n i = n i + g i 2 n_i = n_i +g_i^2

3. 迭代

# Reference

• 《美团机器学习实践》by美团算法团队，第三章
• 《机器学习》by周志华，第三、四章
• 白板推导系列，shuhuai007
10-27 604
02-19 175

02-05 1万+
08-17 576
10-08 3146