Kaggle | Santander Product Recommendation比赛总结

Santander Product Recommendation应该是我参加kaggle的第一场比赛，虽然这场比赛最终只取得了103/1704（TOP 6%）的成绩，不过这场比赛让我熟悉了kaggle这个平台，结交了一些志同道合的kaggler。在持续的两个月的过程中，我进行了很多的尝试，学到了很多，下面对这场比赛进行个小结。

题目：根据西班牙Santander银行前17个月的商品购买记录以及用户属性去预测2016年6月每个用户最可能购买的商品（给出预测的7个商品，按可能性大小排列，评测指标用MAP@7）。

降低电脑使用内存：这题的数据量很大，在两个月的比赛过程中，本博主的电脑死机不下50次。本博主参加kaggle这题后，买了根8g的内存条把自己的电脑内存扩充到14g，可是还是经常会死机（程序写的太复杂或者应用软件开的过多）。处理数据如果读入一行处理一行，内存的占用率不会太大，可是这样对数据进行分组聚合和一些统计的处理会显得很麻烦。所以由于本博主处理能力有限，每次都是将整个表读入内存再进行处理，虽然很占内存，可是预处理提取特征这些操作会相对容易实现。这里用pandas的read_csv把2g的数据读到内存会占用10g的内存，因此我们读数据时可以通过改变数据读入的类型（将float64设置成int8读入）或者只读入需要的列，来减少内存的使用。另外，在编程过程中应及时删除一些不用的表来释放内存。

缺失值处理：这里只想说我在比赛过程中遇到的一个坑，csv中缺失的数据（为空）在采用不同的读入方式会呈现不同的结果。采用csv.DictReader( )会将csv中的缺失值读为' ',而用pd.read_csv( )会将缺失值读为nan。剩下的在缺失值的处理中要注意的就是数据类型的转换。

模型：这题发布的第一个月，各位kaggler尝试了各种方法模型，效果都不是很好。后来在BreakfastPirate发布了一条论坛介绍了自己的思路后（由于6月是商品销售分布和其它月差别很大，因此使用xgboost多分类用训练集里的6月去预测测试集6月，这样也大大减小了训练集），比赛排行榜上各种上升，各位kaggler都在此思路上进行特征工程来提高自己的分数。可能是本博主水平略差，我将自己能想到的特征都尝试了一遍可是分数依然提高不多，在比赛最后7天，我没有任何突破，名次掉了30多，哎，真的不会啊，每次加完特征都没提升，有时还会下降，我特么都开始怀疑人生了。此外，这题ensemble没什么用。

特征工程：lag-5特征，renta等属性分组购买率，各种sum，一些商品组合的求和及编码。

线下验证：本博主参加了两次kaggle了，每次比赛结束由Public LeaderBoard转到Private LeaderBoard时，名次都会相对下降一些，一方面可能是因为模型的鲁棒性不是很好，还有一点原因可能是因为过多依赖线上结果。因此，我们比赛时应多参考线下结果，不能一味的依赖于线上结果，这样可能会过拟合Public LeaderBoard。

其它比赛思路：Breakfast  Pirate

我的比赛核心代码如下：

'''
   author:TaoZI
   date:2016/12/22
'''
import datetime
import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.cross_validation import KFold

pd.options.mode.chained_assignment = None

mapping_dict = {
'sexo'          : {'nan':0,'H':0, 'V':1},
'ind_actividad_cliente' : {'nan':0, '0.0':0, '0':0,'1.0':1, '1':1},
'segmento'      : {'nan':0, '01 - TOP':0, '03 - UNIVERSITARIO':1, '02 - PARTICULARES':2},
'ind_nuevo'     : {'nan':0, '1.0':0, '1':0,  '0.0':1, '0':1 },
'tiprel_1mes'   : {'nan':0, 'P':0, 'R':0, 'N':0, 'I':1, 'A':2},
'indext'        : {'nan':0, 'S':0, 'N':1}
}

target_raw_cols = ['ind_ahor_fin_ult1', 'ind_aval_fin_ult1', 'ind_cco_fin_ult1',  'ind_cder_fin_ult1',
                   'ind_cno_fin_ult1',  'ind_ctju_fin_ult1', 'ind_ctma_fin_ult1', 'ind_ctop_fin_ult1',
                   'ind_ctpp_fin_ult1', 'ind_deco_fin_ult1', 'ind_deme_fin_ult1', 'ind_dela_fin_ult1',
                   'ind_ecue_fin_ult1', 'ind_fond_fin_ult1', 'ind_hip_fin_ult1',  'ind_plan_fin_ult1',
                   'ind_pres_fin_ult1', 'ind_reca_fin_ult1', 'ind_tjcr_fin_ult1', 'ind_valo_fin_ult1',
                   'ind_viv_fin_ult1',  'ind_nomina_ult1',   'ind_nom_pens_ult1', 'ind_recibo_ult1']

target_cols = target_raw_cols[2:]

con_cols = ['ncodpers', 'fecha_dato', 'age', 'antiguedad','renta']
cat_cols = mapping_dict.keys()
user_cols = con_cols + cat_cols + target_raw_cols
NUM_CLASS = 22

def getAge(str_age):
    age = str_age.strip()
    if age == 'NA' or age == 'nan':
        age1 = 2
    elif float(age) < 20:
        age1 = 0
    elif float(age) < 30:
        age1 = 1
    elif float(age) < 40:
        age1 = 2
    elif float(age) < 50:
        age1 = 3
    elif float(age) < 60:
        age1 = 4
    else:
        age1 =  5
    return age1

def getCustSeniority(str_seniority):
    cust_seniority = str_seniority.strip()
    if cust_seniority == 'NA' or cust_seniority == 'nan':
        seniority = 4
    elif float(cust_seniority) < 50:
        seniority = 0
    elif float(cust_seniority) < 75:
        seniority = 1
    elif float(cust_seniority) < 100:
        seniority = 2
    elif float(cust_seniority) < 125:
        seniority = 3
    elif float(cust_seniority) < 150:
        seniority = 4
    elif float(cust_seniority) < 175:
        seniority = 5
    elif float(cust_seniority) < 200:
        seniority = 6
    elif float(cust_seniority) < 225:
        seniority = 7
    else:
        seniority = 8
    return seniority

def getRent(str_rent):
    rent = str_rent.strip()
    if rent == 'NA' or rent == 'nan':
        rent1 = 4
    elif float(rent) < 45542.97:
        rent1 = 1
    elif float(rent) < 57629.67:
        rent1 = 2
    elif float(rent) < 68211.78:
        rent1 = 3
    elif float(rent) < 78852.39:
        rent1 = 4
    elif float(rent) < 90461.97:
        rent1 = 5
    elif float(rent) < 103855.23:
        rent1 = 6
    elif float(rent) < 120063.00:
        rent1 = 7
    elif float(rent) < 141347.49:
        rent1 = 8
    elif float(rent) < 173418.12:
        rent1 = 9
    elif float(rent) < 234687.12:
        rent1 = 10
    else:
        rent1 = 11
    return rent1

def add_com_features(lag_feats):

    lag_feats['prod_sum'] = lag_feats.apply(lambda x: np.sum(x[-120:]), axis = 1)

    for i, pre in enumerate(['1_', '2_', '3_', '4_', '5_']):
        pre_cols = [pre + col for col in target_raw_cols]
        lag_feats['sum_24_' + str(i + 1)] = lag_feats.loc[:, pre_cols].sum(axis = 1)
    sum_24_list = ['sum_24_' + str(i + 1) for i in range(5)]
    lag_feats['sum_24_max'] = lag_feats[sum_24_list].max(axis = 1)
    lag_feats['sum_24_min'] = lag_feats[sum_24_list].min(axis = 1)
    lag_feats['sum_24_mean'] = lag_feats[sum_24_list].mean(axis = 1)

    for i, col in enumerate(target_raw_cols):
        index_list = [pre + col for pre in ['1_', '2_', '3_', '4_', '5_']]
        lag_feats['prod_sum_' + str(i)] = lag_feats.loc[:, index_list].sum(axis = 1)

    pro_sum_list = ['prod_sum_' + str(i) for i in range(24)]
    for gp_col in ['renta', 'sexo']:
        group_feats = lag_feats[pro_sum_list].groupby(lag_feats[gp_col]).agg(lambda x: round(x.sum() / x.count(), 2))
        group_feats.columns = [gp_col + str(i) for i in range(24)]
        lag_feats = pd.merge(lag_feats, group_feats, left_on = gp_col, right_index = True, how = 'left')

    com_col = [[0, 2], [7, 8, 9], [9, 10, 11], [19, 20, 21]]
    for x in range(4):
        import_col = [target_cols[i] for i in com_col[x]]
        for i in range(1, 6):
            pre_import_col = [str(i) + '_' + col for col in import_col]
            lag_feats[str(i) + '_' + str(x + 1) + '_s_sum_import'] = lag_feats[pre_import_col].sum(axis = 1)
    return lag_feats

def process_train_data(in_file_name, date_list):

    this_month = in_file_name[in_file_name['fecha_dato'].isin([date_list[0]])]
    for col in cat_cols:
        this_month[col] = this_month[col].apply(lambda x:mapping_dict[col][str(x)])
    for col in  target_raw_cols:
        this_month[col].fillna(0, inplace=True)
    this_month['age'] = this_month['age'].apply(lambda x: getAge(x))
    this_month['antiguedad'] = this_month['antiguedad'].apply(lambda x: getCustSeniority(x))
    this_month['renta'] = this_month['renta'].apply(lambda x: getRent(str(x)))

    hist_data = in_file_name.loc[:,['ncodpers','fecha_dato'] + target_raw_cols]
    del in_file_name
    pre_month = hist_data[hist_data['fecha_dato'].isin([date_list[1]])]
    pre_month_ncodpers = pre_month[['ncodpers']]
    pre_month_target = pre_month[target_raw_cols]
    pre_month_target = pre_month_target.add_prefix('1_')
    pre_month = pd.concat([pre_month_ncodpers, pre_month_target], axis=1)
    this_month = pd.merge(this_month, pre_month, on=['ncodpers'], how='left')
    this_month.fillna(0, inplace=True)
    for col in target_cols:
        this_month[col] = np.where(this_month[col]-this_month['1_'+col] > 0,(this_month[col]-this_month['1_'+col]), 0 )

    this_month_target = this_month[target_cols]
    this_month = this_month.drop(target_raw_cols, axis=1)

    x_vars_list = []
    y_vars_list = []

    for i in range(2, len(date_list)):
        tmp = hist_data[hist_data['fecha_dato'].isin([date_list[i]])].loc[:,['ncodpers'] + target_raw_cols]
        tmp = tmp.add_prefix(str(i) + "_")
        tmp.rename(columns={str(i) + '_ncodpers': 'ncodpers'}, inplace=True)
        this_month = pd.merge(this_month, tmp, on=['ncodpers'], how='left')
    this_month.fillna(0, inplace=True)
    del hist_data

    this_month = add_com_features(this_month)
    this_month.fillna(0, inplace=True)

    this_month = pd.concat([this_month,this_month_target],axis=1)
    for idx,row in this_month.iterrows():
        for i in range(0,22):
            if row[(-22+i)]>0:
                x_vars_list.append(row[:-22])
                y_vars_list.append(i)

    return np.array(x_vars_list), np.array(y_vars_list)

def process_test_data(test_file, hist_file, date_list):
    for col in cat_cols:
        test_file[col] = test_file[col].apply(lambda x: mapping_dict[col][str(x)])
    test_file['age'] = test_file['age'].apply(lambda x: getAge(x))
    test_file['antiguedad'] = test_file['antiguedad'].apply(lambda x: getCustSeniority(x))
    test_file['renta'] = test_file['renta'].apply(lambda x: getRent(x))

    for i in range(0, len(date_list)):
        tmp = hist_file[hist_file['fecha_dato'].isin([date_list[i]])].loc[:,['ncodpers'] + target_raw_cols]
        tmp = tmp.add_prefix(str(i + 1) + "_")
        tmp.rename(columns={str(i + 1) + '_ncodpers': 'ncodpers'}, inplace=True)
        test_file = pd.merge(test_file, tmp, on=['ncodpers'], how='left')
    test_file.fillna(0, inplace=True)

    del hist_file

    test_file = add_com_features(test_file)
    test_file.fillna(0, inplace=True)
    return test_file.values

def runXGB_CV(train_X,train_y, test_X, index, seed_val):

    train_index, test_index = index
    X_train = train_X[train_index]
    y_train = train_y[train_index]

    xgtrain = xgb.DMatrix(X_train, label=y_train)
    xgtest  = xgb.DMatrix(test_X)

    param = {
        'objective' : 'multi:softprob',
        'eval_metric' : "mlogloss",
        'num_class' : NUM_CLASS,
        'silent' : 1,
        'min_child_weight' : 2,
        'eta': 0.05,
        'max_depth': 6,
        'subsample' : 0.9,
        'colsample_bytree' : 0.8,
        'seed' : seed_val
    }
    num_rounds = 100
    model  = xgb.train(param, xgtrain, num_rounds)
    pred   = model.predict(xgtest)
    return pred


def runXGB(train_X, train_y, test_X,seed_val=123):
    param = {
        'objective' : 'multi:softprob',
        'eval_metric' : "mlogloss",
        'num_class' : NUM_CLASS,
        'silent' : 1,
        'min_child_weight' : 2,
        'eta': 0.05,
        'max_depth': 6,
        'subsample' : 0.9,
        'colsample_bytree' : 0.8,
        'seed' : seed_val
    }
    num_rounds = 100
    xgtrain = xgb.DMatrix(train_X, label = train_y)
    xgtest  = xgb.DMatrix(test_X)

    model  = xgb.train(param, xgtrain, num_rounds)
    preds  = model.predict(xgtest)
    return preds


if __name__ == "__main__":

    cv_sel = 1
    start_time = datetime.datetime.now()
    data_path = '../input/'

    print "feature extract..."
    train_file = pd.read_csv(data_path + 'train_ver3.csv',
                             dtype = {'age': 'str', 'antiguedad': 'str', 'renta': 'str'},
                             usecols = user_cols)
    print datetime.datetime.now() - start_time

    train_X, train_y = process_train_data(train_file, ['2015-06-28', '2015-05-28', '2015-04-28',
                                                       '2015-03-28', '2015-02-28', '2015-01-28'])
    train_X = train_X[:, 2:]
    print datetime.datetime.now() - start_time

    data_date = ['2016-05-28', '2016-04-28', '2016-03-28', '2016-02-28', '2016-01-28']
    train_file = train_file[train_file['fecha_dato'].isin(data_date)].loc[:,
                 ['ncodpers', 'fecha_dato'] + target_raw_cols]

    test_file = pd.read_csv(data_path + 'test_ver3.csv',
                            dtype = {'age': 'str', 'antiguedad': 'str', 'renta': 'str'},
                            usecols = con_cols + cat_cols)

    test_X = process_test_data(test_file, train_file, data_date)
    print datetime.datetime.now() - start_time

    del train_file, test_file
    test_X = test_X[:, 2:]
    print train_X.shape, train_y.shape, test_X.shape
    print datetime.datetime.now() - start_time

    seed_val = 123
    if cv_sel == 1:
        print "running model with cv..."
        nfolds = 5
        kf = KFold(train_X.shape[0], n_folds = nfolds, shuffle = True, random_state = seed_val)
        preds = [0] * NUM_CLASS
        for i, index in enumerate(kf):
            preds += runXGB_CV(train_X, train_y, test_X, index, seed_val)
            print 'fold %d' % (i + 1)
        preds = preds / nfolds

    else:
        print "running model with feature..."
        preds = runXGB(train_X, train_y, test_X,seed_val)

    del train_X, test_X, train_y

    print "Getting the top products.."
    target_cols = np.array(target_cols)
    preds = np.argsort(preds, axis = 1)
    preds = np.fliplr(preds)[:, :7]
    test_id = np.array(pd.read_csv( data_path + 'test_ver2.csv', usecols = ['ncodpers'])['ncodpers'])
    final_preds = [" ".join(list(target_cols[pred])) for pred in preds]
    out_df = pd.DataFrame({'ncodpers': test_id, 'added_products': final_preds})
    out_df.to_csv('../submit/sub_xgb.csv', index = False)

GitHub :  https://github.com/wenwu313/Kaggle-Solution