最近报名参加了天池大数据比赛,题目是搭建一个推荐系统,我使用的是Python语言,挖掘的第一步是输入原始数据,折腾了很久才把数据按照期望的格式进行输入存储,在这里介绍一下
- 天池大数据比赛链接: http://tianchi.aliyun.com/competition/information.htm?spm=5176.100067.5678.2.X8Bm6J&raceId=231506
- 数据源分析:数据的原始格式如下,存在两个问题,其一是文件的分隔符不统一,有空格也有逗号,其二是第三列(也就是商品分词后的结果)的分词数量不一致
41 399 131001,58750,195405,23594,131035,43341,92435,18294,43299,43660,86400,25984,59461,91068 46 338 42453,52550,142033,64060,37355,92435,66180,122877,84899,186483,101916,178051,148830,43299,208642 116 396 200256,69355,92435,43341,132321,218736,140701,137354,91068,2110,26749,198110,195405,131001 121 221 19244,2592,77682,183335,18294,131001,154301,156739,25155,40127,195405,37355,44665 166 199 6603,55565,59009,141815,17823,56660,94566,168708,186022,47149,178981,40718,119433,143243 216 627 37355,11054,48817,127829,102013,18928,100780,205795,170954,61410,59606,145333,29836,18136,157099 236 235 198355,92344,92435,128535,165874,37355,104975,69042,202561,218532,103148,13817 246 220 131035,89627,92435,98279,189223,297,11054,116175,56022,78553,164790,86400,195405,34939 281 487 168887,183868,200798,165530,94632,188186,205124,109662,99521,109662,66980,131060,200762
- 处理方法1:首先是在read_csv函数中设定converters参数,将第三列的数据转化成数组进行存储;另外是重新将read_csv获得的DataFrame的第三列进行拆分,重新赋值给新的DataFrame,问题顺利解决
import pandas as pd
def seq_item(s_items):
# 将商品描述词分列,商品最多的商品描述分词数为33个
return s_items.split(',')
def read_dim_items(read_rows = None):
# 读入商品信息表,参数read_rows表示读入的行数,默认是全读
dim_items = pd.read_table('dim_items\\dim_items.txt',header = None,sep = ' ',converters = {2:seq_item},index_col = [1,0],nrows = read_rows)
dim_items = pd.DataFrame(dim_items[2].tolist(),index = dim_items.index)
return dim_items
- 处理方法2:利用pandas自带的str.split函数进行拆分并重新赋值给新的df
def read_dim_items(read_rows = None):
# 读入商品信息表,参数read_rows表示读入的行数,默认是全读
# 第一列为商品ID(item_id),第二列为商品所属类目ID(cat_id),第三列为商品标题分词后的结果(items)
dim_items = pd.read_table('dim_items\\dim_items.txt',header = None,sep = ' ',index_col = [1,0],nrows = read_rows) # ,converters = {2:seq_item}
dim_items = dim_items[2].str.split(',',expand = True)
return dim_items