一、检视原数据集
读入数据并检测
import numpy as np
import pandas as pd
file=open("data/housing/housing.csv")
train_df=pd.read_csv(file)
train_df.head()
longitude | latitude | housing_median_age | total_rooms | total_bedrooms | population | households | median_income | median_house_value | ocean_proximity | |
---|---|---|---|---|---|---|---|---|---|---|
0 | -122.23 | 37.88 | 41.0 | 880.0 | 129.0 | 322.0 | 126.0 | 8.3252 | 452600.0 | NEAR BAY |
1 | -122.22 | 37.86 | 21.0 | 7099.0 | 1106.0 | 2401.0 | 1138.0 | 8.3014 | 358500.0 | NEAR BAY |
2 | -122.24 | 37.85 | 52.0 | 1467.0 | 190.0 | 496.0 | 177.0 | 7.2574 | 352100.0 | NEAR BAY |
3 | -122.25 | 37.85 | 52.0 | 1274.0 | 235.0 | 558.0 | 219.0 | 5.6431 | 341300.0 | NEAR BAY |
4 | -122.25 | 37.85 | 52.0 | 1627.0 | 280.0 | 565.0 | 259.0 | 3.8462 | 342200.0 | NEAR BAY |
每一行都表示一个街区。共有 10 个属性:经度、维度、房屋年龄中位数、总房间数、卧室数量、人口数、家庭数、收入中位数、房屋价值中位数、离大海距离。
train_df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 10 columns):
longitude 20640 non-null float64
latitude 20640 non-null float64
housing_median_age 20640 non-null float64
total_rooms 20640 non-null float64
total_bedrooms 20433 non-null float64
population 20640 non-null float64
households 20640 non-null float64
median_income 20640 non-null float64
median_house_value 20640 non-null float64
ocean_proximity 20640 non-null object
dtypes: float64(9), object(1)
memory usage: 1.6+ MB
可以看出total_bedrooms这一项有缺失值,后面要进行处理。ocean_proximity这一项的数据类型为类别型数据
train_df.ocean_proximity.value_counts()
<1H OCEAN 9136
INLAND 6551
NEAR OCEAN 2658
NEAR BAY 2290
ISLAND 5
Name: ocean_proximity, dtype: int64
value_counts()方法查看都有什么类型,每个类都有多少街区
train_df.describe()
longitude | latitude | housing_median_age | total_rooms | total_bedrooms | population | households | median_income | median_house_value | |
---|---|---|---|---|---|---|---|---|---|
count | 20640.000000 | 20640.000000 | 20640.000000 | 20640.000000 | 20433.000000 | 20640.000000 | 20640.000000 | 20640.000000 | 20640.000000 |
mean | -119.569704 | 35.631861 | 28.639486 | 2635.763081 | 537.870553 | 1425.476744 | 499.539680 | 3.870671 | 206855.816909 |
std | 2.003532 | 2.135952 | 12.585558 | 2181.615252 | 421.385070 | 1132.462122 | 382.329753 | 1.899822 | 115395.615874 |
min | -124.350000 | 32.540000 | 1.000000 | 2.000000 | 1.000000 | 3.000000 | 1.000000 | 0.499900 | 14999.000000 |
25% | -121.800000 | 33.930000 | 18.000000 | 1447.750000 | 296.000000 | 787.000000 | 280.000000 | 2.563400 | 119600.000000 |
50% | -118.490000 | 34.260000 | 29.000000 | 2127.000000 | 435.000000 | 1166.000000 | 409.000000 | 3.534800 | 179700.000000 |
75% | -118.010000 | 37.710000 | 37.000000 | 3148.000000 | 647.000000 | 1725.000000 | 605.000000 | 4.743250 | 264725.000000 |
max | -114.310000 | 41.950000 | 52.000000 | 39320.000000 | 6445.000000 | 35682.000000 | 6082.000000 | 15.000100 | 500001.000000 |
画出每个数值属性的柱状图
%matplotlib inline
from matplotlib import pyplot as plt
plt.style.use('ggplot')
train_df.hist(bins=50,figsize=(16,9))
array([[<matplotlib.axes._subplots.AxesSubplot object at 0x00000248B0C2E3C8>,
<matplotlib.axes._subplots.AxesSubplot object at 0x00000248B3C91FD0>,
<matplotlib.axes._subplots.AxesSubplot object at 0x00000248B3CC1588>],
[<matplotlib.axes._subplots.AxesSubplot object at 0x00000248B3CE9B00>,
<matplotlib.axes._subplots.AxesSubplot object at 0x00000248B3D1B0B8>,
<matplotlib.axes._subplots.AxesSubplot object at 0x00000248B3D41630>],
[<matplotlib.axes._subplots.AxesSubplot object at 0x00000248B3D6ABA8>,
<matplotlib.axes._subplots.AxesSubplot object at 0x00000248B3D98198>,
<matplotlib.axes._subplots.AxesSubplot object at 0x00000248B3D981D0>]],
dtype=object)
从柱状图中可以发现以下问题:
1.这些属性的量度不一样,在后面需要进行特征缩放(参数的各个数值差特别大的时候要进行特征缩放)
2