6.3 python ipython

在ipython notebook上完成

%matplotlib inline

import random

import numpy as np
import scipy as sp
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

import statsmodels.api as sm
import statsmodels.formula.api as smf

sns.set_context("talk")

Anscombe's quartet

Anscombe's quartet comprises of four datasets, and is rather famous. Why? You'll find out in this exercise.

解释：Anscombe四重奏，用4组数据说明了画图的重要性。这四组数据均值，方差均相等，这将导致线性回归的结果与图像完全不符。

Part 1

For each of the four datasets...

• Compute the mean and variance of both x and y
• Compute the correlation coefficient between x and y
• Compute the linear regression line: y=β0+β1x+ϵy=β0+β1x+ϵ (hint: use statsmodels and look at the Statsmodels notebook)

均值和方差：

调用groupby函数聚类，然后调用mean和var函数对每组的x和y求均值和方差

相关性：

没找到好的函数，因此用了循环，获取每组的10个数据，之后对每组求corr。

（这里不知道为什么不能只用groupby函数，可能这个函数返回的对象不是原来的数据吧）

线性回归：

调用statsmodels.api中的OLS函数，但是没找到如何单个输出所求的东西的方法。

print("每组x的均值")
print(anascombe.groupby('dataset')['x'].mean())
print("\n每组x的方差")
print(anascombe.groupby('dataset')['x'].var())
print("\n每组y的均值")
print(anascombe.groupby('dataset')['y'].mean())
print("\n每组y的方差")
print(anascombe.groupby('dataset')['y'].var())


print("\n相关性")
for i in range(4):
    x = anascombe.x[i*10:(i+1)*10]
    y = anascombe.y[i*10:(i+1)*10]
    corrlation = x.corr(y)
    print("corrlation of group", i, ':', corrlation)
    print()

    
print("\n线性回归")
for i in range(4):
    x = anascombe.x[i*10:(i+1)*10]
    y = anascombe.y[i*10:(i+1)*10]
    mod = sm.OLS(y,x)
    result = mod.fit()
    print(result.summary())

结果如下

每组x的均值
dataset
I      9.0
II     9.0
III    9.0
IV     9.0
Name: x, dtype: float64

每组x的方差
dataset
I      11.0
II     11.0
III    11.0
IV     11.0
Name: x, dtype: float64

每组y的均值
dataset
I      7.500909
II     7.500909
III    7.500000
IV     7.500909
Name: y, dtype: float64

每组y的方差
dataset
I      4.127269
II     4.127629
III    4.122620
IV     4.123249
Name: y, dtype: float64

相关性
corrlation of group: 0 0.797081575906253

corrlation of group: 1 0.8107567988514719

corrlation of group: 2 0.828558301914895

corrlation of group: 3 0.4695259621639301


线性回归
                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.965
Model:                            OLS   Adj. R-squared:                  0.962
Method:                 Least Squares   F-statistic:                     251.5
Date:                Sat, 09 Jun 2018   Prob (F-statistic):           6.95e-08
Time:                        17:13:54   Log-Likelihood:                -18.061
No. Observations:                  10   AIC:                             38.12
Df Residuals:                       9   BIC:                             38.43
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
x              0.7881      0.050     15.859      0.000       0.676       0.901
==============================================================================
Omnibus:                        0.651   Durbin-Watson:                   2.507
Prob(Omnibus):                  0.722   Jarque-Bera (JB):                0.396
Skew:                          -0.424   Prob(JB):                        0.820
Kurtosis:                       2.519   Cond. No.                         1.00
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.961
Model:                            OLS   Adj. R-squared:                  0.957
Method:                 Least Squares   F-statistic:                     221.7
Date:                Sat, 09 Jun 2018   Prob (F-statistic):           1.20e-07
Time:                        17:13:54   Log-Likelihood:                -18.584
No. Observations:                  10   AIC:                             39.17
Df Residuals:                       9   BIC:                             39.47
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
x              0.7894      0.053     14.889      0.000       0.669       0.909
==============================================================================
Omnibus:                        3.223   Durbin-Watson:                   2.351
Prob(Omnibus):                  0.200   Jarque-Bera (JB):                1.584
Skew:                          -0.969   Prob(JB):                        0.453
Kurtosis:                       2.795   Cond. No.                         1.00
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.963
Model:                            OLS   Adj. R-squared:                  0.959
Method:                 Least Squares   F-statistic:                     235.0
Date:                Sat, 09 Jun 2018   Prob (F-statistic):           9.34e-08
Time:                        17:13:54   Log-Likelihood:                -18.117
No. Observations:                  10   AIC:                             38.23
Df Residuals:                       9   BIC:                             38.54
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
x              0.8175      0.053     15.329      0.000       0.697       0.938
==============================================================================
Omnibus:                        0.753   Durbin-Watson:                   1.401
Prob(Omnibus):                  0.686   Jarque-Bera (JB):                0.590
Skew:                          -0.489   Prob(JB):                        0.745
Kurtosis:                       2.323   Cond. No.                         1.00
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.964
Model:                            OLS   Adj. R-squared:                  0.960
Method:                 Least Squares   F-statistic:                     243.1
Date:                Sat, 09 Jun 2018   Prob (F-statistic):           8.06e-08
Time:                        17:13:54   Log-Likelihood:                -17.121
No. Observations:                  10   AIC:                             36.24
Df Residuals:                       9   BIC:                             36.54
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
x              0.8537      0.055     15.591      0.000       0.730       0.978
==============================================================================
Omnibus:                        1.048   Durbin-Watson:                   1.199
Prob(Omnibus):                  0.592   Jarque-Bera (JB):                0.714
Skew:                          -0.287   Prob(JB):                        0.700
Kurtosis:                       1.823   Cond. No.                         1.00
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

可以看到均值，方差，相关性都相等。线性回归的结果也相同。从Part2的练习就可以看到图表的重要性了。

Part 2

Using Seaborn, visualize all four datasets.

hint: use sns.FacetGrid combined with plt.scatter

如hint所说的，调用这两个函数即可。但是注意第一个形成Grid的时候需要使用dataset为（标签？）

g = sns.FacetGrid(anascombe, col = 'dataset')
g_map = g.map(plt.scatter, 'x', 'y')

结果如下