一二阶统计量的在线增量计算算法原理推导

一二阶统计量的在线增量计算算法原理推导

统计机器学习领域中经常用到诸如均值，方差，标准差，协方差等统计量；这些统计量频繁计算需要消耗很大的内存，且非常耗时。现有一种常用的在线，增量的统计量的计算方法，可以有效地解决计算大批量数据的统计量时，内存消耗大，耗时较长的问题。

通过算法原理的推导，可以建立统计量的当前值$M_{,k}$与前一个统计量的值$M_{,k-1}$之间的关系（$k$表示待统计的序列的第k个元素），最终实现在线增量的统计量计算算法。

1. 均值 mean

$\bar{x}=\sum _{i=1}^n{x}_i$

2. 方差 variance

$s^2=\frac{\sum _{i=1}^n(x_i-\bar{x}{)}^2}{n-1}=\frac{1}{n(n-1)}[n\sum _{i=1}^n{x}_i^2-(\sum _{i=1}^n{x}_i{)}^2]$

推导公式1：

$s^2=\frac{\sum (x_i-\bar{x}{)}^2}{n-1}$$=\frac{\sum x_i^2-2\bar{x}\sum x_i+\sum {\bar{x}}^2}{n-1}$

$=\frac{\sum x_i^2-2\bar{x}\cdot n\bar{x}+n{\bar{x}}^2}{n-1}$$=\frac{\sum x_i^2-n{\bar{x}}^2}{n-1}$

$=\frac{\sum _{i=1}^n{x}_i^2-\frac{1}{n}\left(\sum _{i=1}^n{x}_i^2\right)}{n-1}$$=\frac{1}{n(n-1)}[n\sum x_i^2-(\sum x_i{)}^2]$

3. 标准差 standard deviation

$s=\sqrt{\frac{\sum _{i=1}^n(x_i-\bar{x}{)}^2}{n-1}}$

4. 协方差 covariance

$cov(x,y)=\frac{\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{n-1}$

推导公式2：

$(n-1)cov(x,y)=\sum (x_i{y}_i-\bar{x}{y}_i-x_i\bar{y}+\bar{x}\bar{y})$$=\sum (x_i{y}_i)-\sum (\bar{x}{y}_i)-\sum (x_i\bar{y})+\sum (\bar{x}\bar{y})$

$=\sum (x_i{y}_i)-n\bar{x}\bar{y}-n\bar{x}\bar{y}+n\bar{x}\bar{y}$$=\sum (x_i{y}_i)-n\bar{x}\bar{y}$

$=\sum (x_i{y}_i-\bar{x}{y}_i)$$=\sum _{i=1}^n{y}_i(x_i-\bar{x})$

则，$cov(x,y)=\frac{1}{n-1}\sum _{i=1}^n{x}_i(y_i-\bar{y})=\frac{1}{n-1}\sum _{i=1}^n{y}_i(x_i-\bar{x})$

M1为一阶累积统计量：$M_1=\sum _{i=1}^k{x}_i$

M2为一阶累积统计量：$M_2=\sum _{i=1}^k{(x_i-\bar{x})}^2$

5. 一二阶统计量的在线增量算法-结论

$M_{1,k}=M_{1,k-1}+(x_k-M_{k-1})/k$

$M_{2,k}=M_{2,k-1}+(x_k-M_{k-1})\ast (x_k-M_k)=M_{2,k-1}+(1-\frac{1}{k})(x_k-M_{1,k-1}{)}^2$

初始条件：$M_1=x_1,$ $M_2=0$

推导过程1：

$M_{1,k}=\sum _{i=1}^k{x}_i=\frac{1}{k}(\sum _{i=1}^k{x}_i+x_k)=\frac{1}{k}(\frac{k-1}{k-1}\sum _{i=1}^{k-1}{x}_i+x_k)$

$=\frac{1}{k}((k-1)M_{1,k-1}+x_k)=\frac{k-1}{k}{M}_{1,k-1}+\frac{1}{k}{x}_k$

$=M_{1,k-1}-\frac{1}{k}{M}_{1,k-1}+\frac{1}{k}{x}_k=M_{1,k-1}+(x_k-M_{1,k-1})/k$

准备两个推导的前提等式：
$x_k-M_{1,k-1}=k(M_{1,k}-M_{1,k-1})$

$m_{1,k-1}=(m_{1,k}-\frac{1}{k}{x}_k)\frac{1}{1-\frac{1}{k}}$

推导过程2：

$M_{2,k}=\sum _{i=1}^k{(x_i-M_{1,k})}^2=\sum _{i=1}^k(x_i-M_{1,k-1}-(x_k-M_{1,k-1})/k{)}^2$

$=[(x_i-M_{1,k-1}{)}^2+\frac{1}{k^2}(x_k-M_{1,k-1}{)}^2-2(x_i-M_{1,k-1})\frac{1}{k}(x_k-M_{1,k-1})]$

$=\sum _{i=1}^k(x_i-M_{1,k-1}{)}^2+\sum _{i=1}^k\frac{1}{k^2}(x_k-M_{1,k-1}{)}^2-\frac{2}{k}\sum _{i=1}^k(x_i-M_{1,k-1})(x_k-M_{1,k-1})$

$=\sum _{i=1}^{k-1}(x_i-M_{1,k-1}{)}^2+(x_k-M_{1,k-1}{)}^2+\frac{1}{k}(x_k-M_{1,k-1}{)}^2-\frac{2}{k}(k{M}_{1,k}-k{M}_{1,k-1})(x_k-M_{1,k-1})$

$=M_{2,k-1}+(x_k-M_{1,k-1}{)}^2+\frac{1}{k}(x_k-M_{1,k-1}{)}^2-2(M_{1,k}-M_{1,k-1})(x_k-M_{1,k-1})$

$=M_{2,k-1}+(x_k-M_{1,k-1}{)}^2+\frac{1}{k}(x_k-M_{1,k-1}{)}^2-2\frac{1}{k}(x_k-M_{1,k-1})(x_k-M_{1,k-1})$

$=M_{2,k-1}+(1-\frac{1}{k})(x_k-M_{1,k-1}{)}^2$

$=M_{2,k-1}+(x_k-M_{1,k-1})(1-\frac{1}{k})(x_k-M_{1,k-1})$

$=M_{2,k-1}+(x_k-M_{1,k-1})[x_k-\frac{1}{k}{x}_k-(1-\frac{1}{k})M_{1,k-1}]$

$=M_{2,k-1}+(x_k-M_{1,k-1})[x_k-\frac{1}{k}{x}_k-(M_{1,k}-\frac{1}{k}{x}_k)]$

$=M_{2,k-1}+(x_k-M_{1,k-1})(x_k-M_{1,k})$

参考资料

1. Skewness - WikiPedia

2. Joanes D N, Gill C A. Comparing measures of sample skewness and kurtosis[J]. Journal of the Royal Statistical Society: Series D (The Statistician), 1998, 47(1): 183-189.

3. binti Yusoff S, Wah Y B. Comparison of conventional measures of skewness and kurtosis for small sample size[C]//2012 International Conference on Statistics in Science, Business and Engineering (ICSSBE). IEEE, 2012: 1-6.

4. Pebay P P. Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments[R]. Sandia National Laboratories, 2008.

5. Online skewness kurtosis computing

6. Online linear regression computing