切分数据集和测试集之前没有注意的事情

分享一下今天遇到的事情

#先将数据集分成训练集和测试集
train_data, test_data = train_test_split(data, test_size=0.2, random_state=7)
'''
剔除掉训练集和测试集中的索引
'''
train_data = train_data.reset_index(drop=True)
test_data = test_data.reset_index(drop=True)


X_train = train_data[['t','ph','pp','re','wc','bd','cc','bc','sc','rp']]
'''
先对训练集进行fit，然后再对训练集进行transform。后面再利用训练集的fit，对测试集进行transform，这样归一化的测试集不会产生数据泄露。
'''
scaler = StandardScaler().fit(X_train)
x_tran=pd.DataFrame(scaler.transform(X_train),columns=['t','ph','pp','re','wc','bd','cc','bc','sc','rp'])
                        

#one-hotting
x_train_ct = pd.get_dummies(train_data['CT'])
x_train_class = pd.get_dummies(train_data['class'])


x_train = pd.concat([x_tran,x_train_ct,x_train_class],axis=1)

y_train = train_data['dmax']
'''
test
'''
X_test = test_data[['t','ph','pp','re','wc','bd','cc','bc','sc','rp']]

x_test_tran=pd.DataFrame(scaler.transform(X_test),columns=['t','ph','pp','re','wc','bd','cc','bc','sc','rp'])
'''
不知道什么原因， pd.get_dummies产生的是false和true，不是0和1，用.astype(int)进行转换
'''
#one-hotting
x_test_ct = pd.get_dummies(test_data['CT']).astype(int)
x_test_class = pd.get_dummies(test_data['class']).astype(int)


x_test = pd.concat([x_test_tran,x_test_ct,x_test_class],axis=1)
y_test = test_data['dmax']