统计系列（四）利用Python进行假设检验

最新推荐文章于 2024-08-03 19:27:22 发布

HsuHeinrich

最新推荐文章于 2024-08-03 19:27:22 发布

阅读量1.7w

点赞数 2

分类专栏：数据分析文章标签： python 数据分析

本文链接：https://blog.csdn.net/weixin_39293132/article/details/128982196

版权

数据分析专栏收录该内容

64 篇文章 41 订阅

订阅专栏

统计系列（四）利用Python进行假设检验

z检验

主要应用场景：在大样本量的总体比例检验
核心：两样本的总体比例差异

单样本比例检验

# 检验样本合格率与0.38是否有差异
import numpy as np
from statsmodels.stats.proportion import proportions_ztest

counts=200; nobs=500; value=0.38

# 计算z检验统计量及p值
proportions_ztest(counts, nobs, value)

(0.9128709291752777, 0.36131042852617834)

双样本比例差异检验

# 两样本合格率是否有差异
import numpy as np
from statsmodels.stats.proportion import proportions_ztest

count1=200; nobs1=500
count2=150; nobs2=500

counts=np.array([count1,count2])
nobs=np.array([nobs1,nobs2])

# 计算z检验统计量及p值
proportions_ztest(counts, nobs)

(3.3149677206589807, 0.0009165370761145276)

t检验

主要应用场景：在小样本量或总体方差未知的情况下，进行单样本或双样本检验
核心：一个两分类自变量与一个连续型因变量。如检验性别（男、女）在薪资上的差异

单样本均值检验

# 检验样本均值与500是否有差异
import pandas as pd
import numpy as np
from scipy import stats as ss

sale = [506,503,489,501,498,497,491,502,490,511,510,504,512,499,487,507,503,488,521,517]
df = pd.DataFrame(sale, columns = ['sale'])

x = np.array(df[['sale']])
mu = np.mean(x)
u0 = 500
print(mu, ss.ttest_1samp(a=x, popmean=u0))

501.8 Ttest_1sampResult(statistic=array([0.83092969]), pvalue=array([0.41633356]))

两独立样本均值差异检验

# 检验两样本均值是否显著差异（不要求样本量一致）
import pandas as pd
import numpy as np
from scipy import stats

a = [145,147,139,138,135,133,135,138,144,143,98,108,109,124,134,129,133,139,141,142,143,145,]
b = [101,98,87,106,105,108,114,112,110,103,105,101,98,87,106,105,108,114,112,110,103,105,101,98]

x = np.array(a)
y = np.array(b)

# 检验两样本是否等方差，不确定前可以用levene检验方差齐性
print(stats.levene(x, y)) # p=0.15不显著,即无法拒绝两样本方差显著差异
      
# 若两样本不具有方差齐性，则需要将equal_val参数设定为False
t, p = stats.ttest_ind(x, y)
print(t, p)

LeveneResult(statistic=2.1445010235453714, pvalue=0.1501903760252867)
9.678388988357316 1.8131877062436242e-12

配对样本均值差检验

# 配对样本校验政策前后是否有显著差异
import pandas as pd
import numpy as np
from scipy.stats import ttest_rel

qian = [88.60,85.20,75.20,78.40,76.00,75.30,89.90,82.70,82.40,75.60,88.60,85.20,75.20,78.40,76.00,75.30,89.90,82.70,82.40,75.60,88.60,85.20,75.20,78.40,76.00,75.30,89.90,82.70,82.40,75.60]
hou = [75.60,76.50,68.20,67.20,69.90,74.30,71.30,78.10,75.30,69.90,75.60,76.50,68.20,67.20,69.90,74.30,71.30,78.10,75.30,69.90,75.60,76.50,68.20,67.20,69.90,74.30,71.30,78.10,75.30,69.90]

dic = {
    
    'qian':qian
    ,'hou':hou
}

df = pd.DataFrame(dic)

x = np.array(df[['qian']])
y = np.array(df[['hou']])
t, p = ttest_rel(x, y)

print(t, p)

[9.5629333] [1.80117663e-10]

卡方检验

主要应用场景：通过分析不同类别数据的相对选择频数和占比情况进行差异判断。

核心：一个多分类自变量与另一个多分类因变量。如检验学历（低、中、高）在收入等级（低、中、高）上的差异

拟合优度检验检验

# 拟合优度检验 观察实验数据与期望数据是否有差异
import pandas as pd
import numpy as np
from scipy import stats
from random import randint
from collections import Counter

# 模拟120次掷骰子
def getData(N, times, sides=6):
    """
    定义函数，获取投掷数据
    N: 表示一次用几个骰子投
    times：表示总共投几次
    sides：表示骰子的面数，默认为6
    """  
    results = []
    for n in range(1,N+1):
        for roll_num in range(times):
            result = randint(1,sides)
            results.append(result)
    return results

roll_result = getData(1, 120)
df = pd.DataFrame(Counter(roll_result).items(), columns=['point', 'cnt']
                     ).sort_values("point")
df['exp'] = 20

# 卡方检验
print(stats.chisquare(df['cnt'], df['exp']))

Power_divergenceResult(statistic=3.3999999999999995, pvalue=0.6385699231037951)

独立性检验

# 独立性检验 构造收入等级与学历列联表
import pandas as pd
import numpy as np
from scipy import stats

# 学历收入数据
ep_dic = {
        '高收入':{'高中及以下':25, '大学':21, '研究生及以上':10},
        '中等收入':{'高中及以下':82, '大学':88, '研究生及以上':30},
        '低收入':{'高中及以下':223, '大学':16, '研究生及以上':5} 
}
df = pd.DataFrame(ep_dic)

print('卡方值=%.4f, p值=%.4f, 自由度=%i 预期频率=%s' % stats.chi2_contingency(df))

卡方值=138.2050, p值=0.0000, 自由度=4 预期频率=[[ 36.96 132.   161.04]
 [ 14.    50.    61.  ]
 [  5.04  18.    21.96]]

方差分析

主要应用场景：分析多个分类变量对连续型因变量的影响。

核心：多个多分类自变量（ $\geq 1$ ）与连续型因变量

单因素方差分析

# 单因素方差分析 学历对收入的影响
import pandas as pd
import numpy as np
from statsmodels.formula.api import ols
from statsmodels.stats.anova import anova_lm

# 学历收入数据
ep_dic = {
        '学历':['A', 'A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B', 'C', 'C', 'C', 'C', 'C', 'C'],
        '收入':[9, 14, 11, 8, 7, 5, 13, 9, 11, 14, 13, 14, 15, 17, 15, 18, 11, 25]
}
df = pd.DataFrame(ep_dic)

formula = '收入~C(学历)'
model = ols(formula=formula, data=df).fit()
anova_lm(model)

	df	sum_sq	mean_sq	F	PR(>F)
C(学历)	2.0	185.444444	92.722222	7.806361	0.004747
Residual	15.0	178.166667	11.877778	NaN	NaN

多因素方差分析

# 多因素方差分析 学历性别对收入的影响
import pandas as pd
import numpy as np
from statsmodels.formula.api import ols
from statsmodels.stats.anova import anova_lm

# 学历收入数据
ep_dic = {
        '学历':['A', 'A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B', 'C', 'C', 'C', 'C', 'C', 'C'],
        '性别':['f', 'f', 'm', 'm', 'm', 'm', 'm', 'm', 'f', 'm', 'f', 'm', 'm', 'f', 'f', 'm', 'm', 'm'],
        '收入':[9, 14, 11, 8, 7, 5, 13, 9, 11, 14, 13, 14, 15, 17, 15, 18, 11, 25]
}
df = pd.DataFrame(ep_dic)

formula = '收入~ 学历 + 性别'
model = ols(formula=formula, data=df).fit()
anova_lm(model)

	df	sum_sq	mean_sq	F	PR(>F)
学历	2.0	185.444444	92.722222	7.359370	0.006542
性别	1.0	1.777778	1.777778	0.141102	0.712818
Residual	14.0	176.388889	12.599206	NaN	NaN

样本方差检验

主要应用场景：检验样本的方差波动

单样本方差检验

# 检验样本方差是否等于1%（95%置信度）
import pandas as pd
import numpy as np

x = np.array([0.564409196,0.264802098,0.947742641,0.276915401,0.118015848,0.40797025,-0.72194916,0.871691048,0.461142898,0.421672612])
n = len(x)
s2 = np.var(x)
s0 = 0.01 # 估计方差基准值
# 计算卡方值
chi2 = (n-1)*s2/s0
print(chi2)

171.94566597969543

双样本方差检验

# 检验两样本的波动是否一致
import pandas as pd
import numpy as np
from scipy import stats
from statsmodels.formula.api import ols
from statsmodels.stats.anova import anova_lm

dic = {
    'x':[0.424155712,0.898346186,0.521924734,0.841409445,0.211007655,0.564409196,0.264802098,0.061063781,0.555238843,0.871449053,0.255149871,0.044518802,0.424155712,0.898346186,0.521924734,0.841409445,0.211007655,0.564409196,0.264802098,0.947742641],
    'y':[0.261074632,0.165020704,0.760604024,0.371380478,0.379540861,0.967873454,0.582328379,0.102436937,0.577388406,0.109178342,0.974608779,0.216238976,0.261074632,0.165020704,0.760604024,0.371380478,0.379540861,0.967873454,0.582328379,0.795299947]
}
df = pd.DataFrame(dic)

formula = 'x~y'
model = ols(formula,df).fit()
results = anova_lm(model)
print(results) # 无法拒绝波动率一致，即波动率显著相同

            df    sum_sq   mean_sq         F   PR(>F)
y          1.0  0.000709  0.000709  0.007744  0.93085
Residual  18.0  1.648029  0.091557       NaN      NaN