增量计算海量数据均值、方差、标准差增
本文转载自博客:
http://www.calmkart.com/?p=369
前言:
最近需要从海量数据中获取均值,方差,标准差
显然直接读如内存中按公式做肯定是gg的,于是考虑是否可以增量计算
最终写了一个增量计算海量数据均值方差标准差的python通用类
参考了以下公式推导:
代码如下:
# -*- coding: utf-8 -*-
from __future__ import division
import numpy
class incre_std_avg():
'''
增量计算海量数据平均值和标准差,方差
1.数据
obj.avg为平均值
obj.std为标准差
obj.n为数据个数
对象初始化时需要指定历史平均值,历史标准差和历史数据个数(初始数据集为空则可不填写)
2.方法
obj.incre_in_list()方法传入一个待计算的数据list,进行增量计算,获得新的avg,std和n(海量数据请循环使用该方法)
obj.incre_in_value()方法传入一个待计算的新数据,进行增量计算,获得新的avg,std和n(海量数据请将每个新参数循环带入该方法)
'''
def __init__(self, h_avg=0, h_std=0, n=0):
self.avg = h_avg
self.std = h_std
self.n = n
def incre_in_list(self, new_list):
avg_new = numpy.mean(new_list)
incre_avg = (self.n*self.avg+len(new_list)*avg_new) / \
(self.n+len(new_list))
std_new = numpy.std(new_list, ddof=1)
incre_std = numpy.sqrt((self.n*(self.std**2+(incre_avg-self.avg)**2)+len(new_list)
* (std_new**2+(incre_avg-avg_new)**2))/(self.n+len(new_list)))
self.avg = incre_avg
self.std = incre_std
self.n += len(new_list)
def incre_in_value(self, value):
incre_avg = (self.n*self.avg+value)/(self.n+1)
incre_std = numpy.sqrt((self.n*(self.std**2+(incre_avg-self.avg)
** 2)+(incre_avg-value)**2)/(self.n+1))
self.avg = incre_avg
self.std = incre_std
self.n += 1
if __name__ == "__main__":
c = incre_std_avg()
c.incre_in_value(0.05)
print c.avg
print c.std
print c.n
c.incre_in_value(0.02)
c.incre_in_list([0.5, 0.2, 0.3])
print c.avg
print c.std
print c.n
其他参考资料:
- https://blog.csdn.net/zdy0_2004/article/details/46822685
- https://www.cnblogs.com/June2005/p/11498392.html
- 关于np.std的使用,参数ddof需注意,参考;