（2021年3月）Datawhale数据挖掘挑战赛TASK1 baseline

最新推荐文章于 2023-06-27 22:13:47 发布

林立可

最新推荐文章于 2023-06-27 22:13:47 发布

阅读量249

点赞数

分类专栏：机器学习文章标签：数据挖掘机器学习 python

原文链接：https://github.com/datawhalechina/team-learning-data-mining/tree/master/HeartbeatClassification

版权

机器学习专栏收录该内容

5 篇文章 4 订阅

订阅专栏

参加Datawhale数据挖掘挑战赛——零基础入门数据挖掘（心跳信号分类）的学习笔记
比赛地址:https://github.com/datawhalechina/team-learning-data-mining/tree/master/HeartbeatClassification

import os
import gc
import math

import pandas as pd
import numpy as np

import lightgbm as lgb
import xgboost as xgb
from catboost import CatBoostRegressor
from sklearn.linear_model import SGDRegressor, LinearRegression, Ridge
from sklearn.preprocessing import MinMaxScaler

from sklearn.model_selection import StratifiedKFold, KFold
from sklearn.metrics import log_loss
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

from tqdm import tqdm
import matplotlib.pyplot as plt
import time
import warnings
warnings.filterwarnings('ignore')

path = '../data/tianchi/'
# .read_csv返回DataFrame形式的数据
train = pd.read_csv(path+'train.csv')
test = pd.read_csv(path+'testA.csv')
print('Train data shape:',train.shape)
print('TestA data shape:',test.shape)

Train data shape: (100000, 3)
TestA data shape: (20000, 2)

train.head()

	id	heartbeat_signals	label
0	0	0.9912297987616655,0.9435330436439665,0.764677...	0.0
1	1	0.9714822034884503,0.9289687459588268,0.572932...	0.0
2	2	1.0,0.9591487564065292,0.7013782792997189,0.23...	2.0
3	3	0.9757952826275774,0.9340884687738161,0.659636...	0.0
4	4	0.0,0.055816398940721094,0.26129357194994196,0...	2.0

def reduce_mem_usage(df):
    start_mem = df.memory_usage().sum() / 1024**2 
    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))
    
    for col in df.columns:
        col_type = df[col].dtype
        
        if col_type != object:
            c_min = df[col].min()
            c_max = df[col].max()
            if str(col_type)[:3] == 'int':
                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
                    df[col] = df[col].astype(np.int8)
                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                    df[col] = df[col].astype(np.int16)
                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                    df[col] = df[col].astype(np.int32)
                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                    df[col] = df[col].astype(np.int64)  
            else:
                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                    df[col] = df[col].astype(np.float16)
                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                    df[col] = df[col].astype(np.float32)
                else:
                    df[col] = df[col].astype(np.float64)
        else:
            df[col] = df[col].astype('category')

    end_mem = df.memory_usage().sum() / 1024**2 
    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))
    
    return df

# 简单预处理

# 1,新建训练列表train_list
train_list = []

# 2,遍历train数据集
for items in train.values:
    
# 将3列的标签联合拼接到列表中，其中对第2列标签进行了“,”分割操作
train_list.append([items[0]] + [float(i) for i in items[1].split(',')] + [items[2]])

# 3,将列表转为np.array,再用pd.DataFrame转为DataFrame形式的二维数组
train = pd.DataFrame(np.array(train_list))

# 4,设置DataFrame的列格式（可拼接），其中对第2列标签格式做了指定"s_index"，其中index范围是0-（len-2），减去了id和lable
train.columns = ['id'] + ['s_'+str(i) for i in range(len(train_list[0])-2)] + ['label']

# 5,调用方法处理DataFrame类型的train
train = reduce_mem_usage(train)

# 新建测试列表test_list
# 操作同上
test_list=[]
for items in test.values:
    test_list.append([items[0]] + [float(i) for i in items[1].split(',')])

test = pd.DataFrame(np.array(test_list))
test.columns = ['id'] + ['s_'+str(i) for i in range(len(test_list[0])-1)]
test = reduce_mem_usage(test)

Memory usage of dataframe is 157.93 MB
Memory usage after optimization is: 39.67 MB
Decreased by 74.9%
Memory usage of dataframe is 31.43 MB
Memory usage after optimization is: 7.90 MB
Decreased by 74.9%

train.head()

	id	s_0	s_1	s_2	s_3	s_4	s_5	s_6	s_7	s_8	...	label
0	0.0	0.991211	0.943359	0.764648	0.618652	0.379639	0.190796	0.040222	0.026001	0.031708	...	0.0
1	1.0	0.971680	0.929199	0.572754	0.178467	0.122986	0.132324	0.094421	0.089600	0.030487	...	0.0
2	2.0	1.000000	0.958984	0.701172	0.231812	0.000000	0.080688	0.128418	0.187500	0.280762	...	2.0
3	3.0	0.975586	0.934082	0.659668	0.249878	0.237061	0.281494	0.249878	0.249878	0.241455	...	0.0
4	4.0	0.000000	0.055817	0.261230	0.359863	0.433105	0.453613	0.499023	0.542969	0.616699	...	2.0

5 rows × 207 columns

test.head()

	id	s_0	s_1	s_2	s_3	s_4	s_5	s_6	s_7	s_8	...	s_195	s_196	s_197	s_198	s_199	s_200	s_201	s_202	s_203	s_204
0	100000.0	0.991699	1.000000	0.631836	0.136230	0.041412	0.102722	0.120850	0.123413	0.107910	...	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000
1	100001.0	0.607422	0.541504	0.340576	0.000000	0.090698	0.164917	0.195068	0.168823	0.198853	...	0.389893	0.386963	0.367188	0.364014	0.360596	0.357178	0.350586	0.350586	0.350586	0.36377
2	100002.0	0.975098	0.670898	0.686523	0.708496	0.718750	0.716797	0.720703	0.701660	0.596680	...	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000
3	100003.0	0.995605	0.916992	0.520996	0.000000	0.221802	0.404053	0.490479	0.527344	0.518066	...	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000
4	100004.0	1.000000	0.888184	0.745605	0.531738	0.380371	0.224609	0.091125	0.057648	0.003914	...	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000

5 rows × 206 columns

# 使用drop，删除id和label所在列数据,*得到训练数据特征*
x_train = train.drop(['id','label'], axis=1)

# *得到训练数据标签*
y_train = train['label']

# 测试集同上
x_test=test.drop(['id'], axis=1)

# 定义损失函数
def abs_sum(y_pre,y_tru):
    y_pre=np.array(y_pre)
    y_tru=np.array(y_tru)
    loss=sum(sum(abs(y_pre-y_tru)))
    return loss

def cv_model(clf, train_x, train_y, test_x, clf_name):
    folds = 5
    seed = 2021
    
    # *1,数据索引切分配置*
    # sklearn中的模块KFold用于数据索引切分,n_splits表示划分块数,shuffle是否打乱,random_state固定的随机起点
    
    kf = KFold(n_splits=folds, shuffle=True, random_state=seed)
    test = np.zeros((test_x.shape[0],4))

    cv_scores = []
    onehot_encoder = OneHotEncoder(sparse=False)
    
    # *2,交叉验证*
    # enumerate 可以在迭代时绑定元素对应的序号
    # (train_index, valid_index)顾名思义
    for i, (train_index, valid_index) in enumerate(kf.split(train_x, train_y)):
        print('************************************ {} ************************************'.format(str(i+1)))
        # iloc数据切分,切分了特征数据,没有处理标签
        trn_x, trn_y, val_x, val_y = train_x.iloc[train_index], train_y[train_index], train_x.iloc[valid_index], train_y[valid_index]
        
        if clf_name == "lgb":
            # 数据集划分
            train_matrix = clf.Dataset(trn_x, label=trn_y)
            valid_matrix = clf.Dataset(val_x, label=val_y)

            params = {
                'boosting_type': 'gbdt',
                'objective': 'multiclass',
                'num_class': 4,
                'num_leaves': 2 ** 5,
                'feature_fraction': 0.8, #迭代中随机选择80%参数建树
                'bagging_fraction': 0.8, #迭代中使用数据比例
                'bagging_freq': 4,
                'learning_rate': 0.1, #学习率，次轮训练结果度量低于此值将停止训练
                'seed': seed,
                'nthread': 28,
                'n_jobs':24,
                'verbose': -1,
            }

            model = clf.train(params, 
                      train_set=train_matrix, 
                      valid_sets=valid_matrix, 
                      num_boost_round=2000, 
                      verbose_eval=100, 
                      early_stopping_rounds=200)
            val_pred = model.predict(val_x, num_iteration=model.best_iteration)
            test_pred = model.predict(test_x, num_iteration=model.best_iteration) 
            
        val_y=np.array(val_y).reshape(-1, 1)
        val_y = onehot_encoder.fit_transform(val_y)
        print('预测的概率矩阵为：')
        print(test_pred)
        test += test_pred
        score=abs_sum(val_y, val_pred)
        cv_scores.append(score)
        print(cv_scores)
    print("%s_scotrainre_list:" % clf_name, cv_scores)
    print("%s_score_mean:" % clf_name, np.mean(cv_scores))
    print("%s_score_std:" % clf_name, np.std(cv_scores))
#     输出各模块预测和的均值
    test=test/kf.n_splits

    return test

def lgb_model(x_train, y_train, x_test):
    lgb_test = cv_model(lgb, x_train, y_train, x_test, "lgb")
    return lgb_test

lgb_test = lgb_model(x_train, y_train, x_test)

************************************ 1 ************************************
[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24
Training until validation scores don't improve for 200 rounds
[100]	valid_0's multi_logloss: 0.0525735
[200]	valid_0's multi_logloss: 0.0422444
[300]	valid_0's multi_logloss: 0.0407076
[400]	valid_0's multi_logloss: 0.0420398
Early stopping, best iteration is:
[289]	valid_0's multi_logloss: 0.0405457
预测的概率矩阵为：
[[9.99969791e-01 2.85197261e-05 1.00341946e-06 6.85357631e-07]
 [7.93287264e-05 7.69060914e-04 9.99151590e-01 2.00810971e-08]
 [5.75356884e-07 5.04051497e-08 3.15322414e-07 9.99999059e-01]
 ...
 [6.79267940e-02 4.30206297e-04 9.31640185e-01 2.81516302e-06]
 [9.99960477e-01 3.94098074e-05 8.34030725e-08 2.94638661e-08]
 [9.88705846e-01 2.14081630e-03 6.67418381e-03 2.47915423e-03]]
[607.0736049372185]
************************************ 2 ************************************
[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24
Training until validation scores don't improve for 200 rounds
[100]	valid_0's multi_logloss: 0.0566626
[200]	valid_0's multi_logloss: 0.0450852
[300]	valid_0's multi_logloss: 0.044078
[400]	valid_0's multi_logloss: 0.0455546
Early stopping, best iteration is:
[275]	valid_0's multi_logloss: 0.0437793
预测的概率矩阵为：
[[9.99991401e-01 7.69109547e-06 6.65504756e-07 2.42084688e-07]
 [5.72380482e-05 1.32812809e-03 9.98614607e-01 2.66534396e-08]
 [2.82123411e-06 4.13195205e-07 1.34026965e-06 9.99995425e-01]
 ...
 [6.96398024e-02 6.52459907e-04 9.29685742e-01 2.19960932e-05]
 [9.99972366e-01 2.75069005e-05 7.68142933e-08 5.07415018e-08]
 [9.67263676e-01 7.26154408e-03 2.41533542e-02 1.32142531e-03]]
[607.0736049372185, 623.4313863731124]
************************************ 3 ************************************
[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24
Training until validation scores don't improve for 200 rounds
[100]	valid_0's multi_logloss: 0.0498722
[200]	valid_0's multi_logloss: 0.038028
[300]	valid_0's multi_logloss: 0.0358066
[400]	valid_0's multi_logloss: 0.0361478
[500]	valid_0's multi_logloss: 0.0379597
Early stopping, best iteration is:
[340]	valid_0's multi_logloss: 0.0354344
预测的概率矩阵为：
[[9.99972032e-01 2.62406774e-05 1.17282152e-06 5.54230651e-07]
 [1.05242811e-05 6.50215805e-05 9.99924453e-01 6.93812546e-10]
 [1.93240868e-06 1.10384984e-07 3.76773426e-07 9.99997580e-01]
 ...
 [1.34894410e-02 3.84569683e-05 9.86471555e-01 5.46564350e-07]
 [9.99987431e-01 1.25532882e-05 1.03902298e-08 5.46727770e-09]
 [9.78722948e-01 1.06329839e-02 6.94192038e-03 3.70214810e-03]]
[607.0736049372185, 623.4313863731124, 508.02381607269535]
************************************ 4 ************************************
[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24
Training until validation scores don't improve for 200 rounds
[100]	valid_0's multi_logloss: 0.0564768
[200]	valid_0's multi_logloss: 0.0448698
[300]	valid_0's multi_logloss: 0.0446719
[400]	valid_0's multi_logloss: 0.0470399
Early stopping, best iteration is:
[250]	valid_0's multi_logloss: 0.0438853
预测的概率矩阵为：
[[9.99979692e-01 1.70821979e-05 1.27048476e-06 1.95571841e-06]
 [5.66207785e-05 4.02275314e-04 9.99541086e-01 1.82828519e-08]
 [2.62267451e-06 3.58613522e-07 4.78645006e-06 9.99992232e-01]
 ...
 [4.56636552e-02 5.69497433e-04 9.53758468e-01 8.37980573e-06]
 [9.99896785e-01 1.02796802e-04 2.46636563e-07 1.72061021e-07]
 [8.70911669e-01 1.73790185e-02 1.04478175e-01 7.23113697e-03]]
[607.0736049372185, 623.4313863731124, 508.02381607269535, 660.4867407547266]
************************************ 5 ************************************
[LightGBM] [Warning] num_threads is set with nthread=28, will be overridden by n_jobs=24. Current value: num_threads=24
Training until validation scores don't improve for 200 rounds
[100]	valid_0's multi_logloss: 0.0506398
[200]	valid_0's multi_logloss: 0.0396422
[300]	valid_0's multi_logloss: 0.0381065
[400]	valid_0's multi_logloss: 0.0390162
[500]	valid_0's multi_logloss: 0.0414986
Early stopping, best iteration is:
[324]	valid_0's multi_logloss: 0.0379497
预测的概率矩阵为：
[[9.99993352e-01 6.02902202e-06 1.13002685e-07 5.06277302e-07]
 [1.03959552e-05 5.03778956e-04 9.99485820e-01 5.07638601e-09]
 [1.92568065e-07 5.07155306e-08 4.94690856e-08 9.99999707e-01]
 ...
 [8.83103121e-03 2.51969353e-05 9.91142776e-01 9.96143937e-07]
 [9.99984791e-01 1.51997858e-05 5.62426491e-09 3.80450197e-09]
 [9.86084001e-01 8.75968498e-04 1.09742304e-02 2.06580027e-03]]
[607.0736049372185, 623.4313863731124, 508.02381607269535, 660.4867407547266, 539.2160054696064]
lgb_scotrainre_list: [607.0736049372185, 623.4313863731124, 508.02381607269535, 660.4867407547266, 539.2160054696064]
lgb_score_mean: 587.6463107214719
lgb_score_std: 55.944536405714565

# 输出为array类型的结果
lgb_test

array([[9.99981254e-01, 1.71125438e-05, 8.45046636e-07, 7.88733736e-07],
       [4.28215579e-05, 6.13652971e-04, 9.99343511e-01, 1.41575174e-08],
       [1.62884845e-06, 1.96662878e-07, 1.37365693e-06, 9.99996801e-01],
       ...,
       [4.11101448e-02, 3.43163508e-04, 9.58539745e-01, 6.94675406e-06],
       [9.99960370e-01, 3.94933168e-05, 8.45736848e-08, 5.23076338e-08],
       [9.58337628e-01, 7.65806626e-03, 3.06443728e-02, 3.35993298e-03]])

# 转换为DataFrame类型
temp=pd.DataFrame(lgb_test)
temp

	0	1	2	3
0	0.999981	1.711254e-05	8.450466e-07	7.887337e-07
1	0.000043	6.136530e-04	9.993435e-01	1.415752e-08
2	0.000002	1.966629e-07	1.373657e-06	9.999968e-01
3	0.999970	1.909713e-05	1.097002e-05	3.576703e-08
4	0.999983	1.769712e-06	1.482817e-05	1.966254e-07
...	...	...	...	...
19995	0.998096	3.060176e-04	1.085313e-04	1.489757e-03
19996	0.999846	1.436305e-04	1.074898e-05	8.837766e-08
19997	0.041110	3.431635e-04	9.585397e-01	6.946754e-06
19998	0.999960	3.949332e-05	8.457368e-08	5.230763e-08
19999	0.958338	7.658066e-03	3.064437e-02	3.359933e-03

20000 rows × 4 columns

# 保存结果数据到submit.csv
# 这里只是使用了sample_submit的样表

result=pd.read_csv(path+'sample_submit.csv')
result['label_0']=temp[0]
result['label_1']=temp[1]
result['label_2']=temp[2]
result['label_3']=temp[3]
result.to_csv('submit.csv',index=False)

交叉验证kFlod参考：https://www.cnblogs.com/loubin/p/11305565.html
LightGBM参考：https://blog.csdn.net/weixin_39807102/article/details/81912566

林立可

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
（2021年3月）Datawhale数据挖掘挑战赛TASK1 baseline

参加Datawhale数据挖掘挑战赛——零基础入门数据挖掘（心跳信号分类）的学习笔记比赛地址:https://github.com/datawhalechina/team-learning-data-mining/tree/master/HeartbeatClassificationimport osimport gcimport mathimport pandas as pdimport numpy as npimport lightgbm as lgbimport xgboost a
复制链接

扫一扫