数据预处理系列：（一）从外部源获取样本数据

声明：版权所有，转载请联系作者并注明出处  http://blog.csdn.net/u013719780?viewmode=contents

博主简介：风雪夜归子（英文名: Allen），机器学习算法攻城狮，喜爱钻研Machine Learning的黑科技，对Deep Learning和Artificial Intelligence充满兴趣，经常关注kaggle数据挖掘竞赛平台，对数据、Machine Learning和Artificial Intelligence有兴趣的各位童鞋可以一起探讨哦，个人CSDN博客： http://blog.csdn.net/u013719780?viewmode=contents

从外部源获取样本数据

如果条件允许，学本系列内容时尽量用你熟悉的数据集；方便起见，我们用scikit-learn的内置数据库。这些内置数据库可用于测试不同的建模技术，如回归和分类。而且这些内置数据库都是非常著名的数据库。这对不同领域的学术论文的作者们来说是很用的，他们可以用这些内置数据库将他们的模型与其他模型进行比较。

推荐使用IPython来运行文中的指令。大内存很重要，这样可以让普通的命令正常运行。如果用IPython Notebook就更好了。如果你用Notebook，记得用%matplotlib inline指令，这样图象就会出现在Notebook里面，而不是一个新窗口里。

Getting ready

scikit-learn的内置数据库在datasets模块里。用如下命令导入：

In [1]:

from sklearn import datasets
import numpy as np

在IPython里面运行datasets.*?就会看到datasets模块的指令列表。

How to do it…

datasets模块主要有两种数据类型。较小的测试数据集在sklearn包里面，可以通过datasets.load_*?查看。较大的数据集可以根据需要下载。后者默认情况下不在sklearn包里面；但是，有时这些大数据集可以更好的测试模型和算法，因为比较复杂足以模拟现实情形。

默认在sklearn包里面的数据集可以通过datasets.load_*?查看。另外一些数据集需要通过datasets.fetch_*?下载，这些数据集更大，没有被自动安装。经常用于测试那些解决实际问题的算法。

首先，加载boston数据集看看：

In [3]:

boston = datasets.load_boston()
print(boston.DESCR)

Boston House Prices dataset

Notes
------
Data Set Characteristics:  

    :Number of Instances: 506 

    :Number of Attributes: 13 numeric/categorical predictive
    
    :Median Value (attribute 14) is usually the target

    :Attribute Information (in order):
        - CRIM     per capita crime rate by town
        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
        - INDUS    proportion of non-retail business acres per town
        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
        - NOX      nitric oxides concentration (parts per 10 million)
        - RM       average number of rooms per dwelling
        - AGE      proportion of owner-occupied units built prior to 1940
        - DIS      weighted distances to five Boston employment centres
        - RAD      index of accessibility to radial highways
        - TAX      full-value property-tax rate per $10,000
        - PTRATIO  pupil-teacher ratio by town
        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
        - LSTAT    % lower status of the population
        - MEDV     Median value of owner-occupied homes in $1000's

    :Missing Attribute Values: None

    :Creator: Harrison, D. and Rubinfeld, D.L.

This is a copy of UCI ML housing dataset.
http://archive.ics.uci.edu/ml/datasets/Housing


This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
prices and the demand for clean air', J. Environ. Economics & Management,
vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
...', Wiley, 1980.   N.B. Various transformations are used in the table on
pages 244-261 of the latter.

The Boston house-price data has been used in many machine learning papers that address regression
problems.   
     
**References**

   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.
   - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)

DESCR将列出数据集的一些概况。下面我们来下载一个数据集：

In [4]:

housing = datasets.fetch_california_housing()
print(housing.DESCR)

downloading Cal. housing from http://lib.stat.cmu.edu/modules.php?op=modload&name=Downloads&file=index&req=getit&lid=83 to C:\Users\tj2\scikit_learn_data
California housing dataset.

The original database is available from StatLib

    http://lib.stat.cmu.edu/

The data contains 20,640 observations on 9 variables.

This dataset contains the average house value as target variable
and the following input variables (features): average income,
housing average age, average rooms, average bedrooms, population,
average occupation, latitude, and longitude in that order.

References
----------

Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,
Statistics and Probability Letters, 33 (1997) 291-297.

How it works…

当这些数据集被加载时，它们并不是直接转换成Numpy数组。它们是Bunch类型。Bunch是Python常用的数据结构。基本可以看成是一个词典，它的键被实例对象作为属性使用。

用data属性连接数据中包含自变量的Numpy数组，用target属性连接数据中的因变量。

In [5]:

X, y = boston.data, boston.target

网络上Bunch对象有不同的实现；自己写一个也不难。scikit-learn用基本模块定义Bunch。

There's more…

让你从外部源获取数据集时，它默认会被当前文件夹的scikit_learn_data/放在里面，可以通过两种方式进行配置：

• 设置SCIKIT_LEARN_DATA环境变量指定下载位置
• fetch_*?方法的第一个参数是data_home，可以知道下载位置

通过datasets.get_data_home()很容易检查默认下载位置。

See also

UCI机器学习库（UCI Machine Learning Repository）是找简单数据集的好地方。很多scikit-learn的数据集都在那里，那里还有更多的数据集。其他数据源还是著名的KDD和Kaggle。