python 数据预处理

观察数据

1. housing.info()
2. housing.describe()
3. housing.hist(bins=50, figsize=(20,15)) # 连续数据 plt.show()

离散变量使用value_counts()观察：
housing[‘ocean_proximity’].value_counts()

分割数据为测试数据和训练数据
方法1 train_test_split
from sklearn.model_selection import train_test_split
train_set,test_set = train_test_split(housing,test_size=0.2,random_state=42)
方法2 *** StratifiedShuffleSplit
如考虑分层抽样，例如对收入中位数分层抽样（收入中位数对房屋价格比较重要属性）
对中位数分层，ceil取整，得到离散类别。大于5合并为5
housing[‘income_cat’]=np.ceil(housing[‘median_income’]/1.5)
housing[‘income_cat’].where(housing[‘income_cat’]<5,5.0,inplace=True)
使用sklearn StratifiedShuffleSplit***
from sklearn.model_selection import StratifiedShuffleSplit
split = StratifiedShuffleSplit(n_splits=1,test_size=0.2,random_state=42)
for train_index , test_index in split.split(housing,housing[‘income_cat’]):
strat_train_set = housing.loc[train_index]
strat_test_set = housing.loc[test_index]

def income_cat_proportions(data):
return data[“income_cat”].value_counts() / len(data)

train_set, test_set = train_test_split(housing, test_size=0.2, random_state=42)
验证分层抽样
compare_props = pd.DataFrame({
“Overall”: income_cat_proportions(housing),
“Stratified”: income_cat_proportions(strat_test_set),
“Random”: income_cat_proportions(test_set),
}).sort_index()
compare_props[“Rand. %error”] = 100 * compare_props[“Random”] / compare_props[“Overall”] - 100
compare_props[“Strat. %error”] = 100 * compare_props[“Stratified”] / compare_props[“Overall”] - 100
drop [‘income_cat’] column
for set_ in (strat_train_set, strat_test_set):
set_.drop(“income_cat”, axis=1, inplace=True)
使用图表查看相关性longitude and latitude
housing.plot(kind=‘scatter’,x=‘longitude’,y=‘latitude’,alpha=0.1)
加入颜色
housing.plot(kind=“scatter”, x=“longitude”, y=“latitude”, alpha=0.4,
s=housing[“population”]/100, label=“population”, figsize=(10,7),
c=“median_house_value”, cmap=plt.get_cmap(“jet”), colorbar=True,
sharex=False)
plt.legend()
寻找相关性

1.方法1 corr

corr_matrix = housing.corr()
corr_matrix[‘median_house_value’].sort_values(ascending=False)

2. 方法2 scatter_matrix

from pandas.plotting import scatter_matrix

attribues = [‘median_house_value’,‘median_