Pandas学习笔记

摘要

最近在弄腾讯的广告大赛，使用了Pandas作为数据处理的工具，在此记录下使用的方法，以作分享。

参考文献：http://jingyan.baidu.com/season/43456?pn=0

R语言教程：https://edu.aliyun.com/course/27/lesson/list

---

安装

---

功能

读取csv文件

df = pd.read_csv('./original_data/user_installedapps.csv',index_col='userID')

index_col用于指定索引列

groupby

以下几行代码实现类似于数据库里group by后计数的功能，数据库的同功能SQL语句如下

select userID, count(*) from user_installedapps group by userID;

！ 注意 ： groupby的列必须是索引列

df = pd.read_csv('./original_data/user_installedapps.csv',index_col='userID')

grouped_userid = df.groupby(level='userID')

print grouped_userid.count()

//若是要统计总和可以改成sum()
//print grouped_userid.sum()

下面是我写的API，需要的看着改吧

import pandas as pd

conf = {
    # The path of input csv file
    'input_file_path': './original_data/user_installedapps.csv',

    # The index of DataFrame object, which essential for the use of groupby() function
    'index_col': ['userID'],

    # The col which required to by grouped by
    'group_by_col': ['userID'],

    # The path of output csv file
    'output_file_path': r'./processed_data/user_installedapps_count.csv'
}

# <1> File Import and Preparation *****************************
print 'Importing File'

df = pd.read_csv(conf['input_file_path'], index_col=conf['index_col'])

grouped_df = df.groupby(level=conf['index_col'])

print 'Importing File Finished'


# <2> Data Processing Period  *********************************
# The function could be changed as required
print 'Data Processing'

grouped_df_count = grouped_df.count()
# grouped_df_sum = grouped_df.sum()

print 'Data Processing Finished'


# <3> File Export  ********************************************
print 'Exporting File'

grouped_df_count.to_csv(conf['output_file_path'], encoding='gbk')
# grouped_df_sum.to_csv(conf['output_file_path'], encoding='gbk')

print 'Exporting File Finished'

合并csv文件

#coding:utf-8
import csv
import pandas as pd

conf = {
    'base_path': './original_data'
}

# <1> File Import and Preparation *****************************
print 'Importing File'

train          = pd.read_csv(conf['base_path'] + '/train.csv')
ad             = pd.read_csv(conf['base_path'] + '/ad.csv')
app_categories = pd.read_csv(conf['base_path'] + '/app_categories.csv')
position       = pd.read_csv(conf['base_path'] + '/position.csv')
test           = pd.read_csv(conf['base_path'] + '/test.csv')
user           = pd.read_csv(conf['base_path'] + '/user.csv')
# user_app_actions = pd.read_csv('./data/csv/user_app_actions.csv')
# user_installedapps = pd.read_csv('./data/csv/user_installedapps.csv')

print 'Importing File Finished'



# <2> Data Processing Period  *********************************
# The function could be changed as required
print 'Data Processing'

data = pd.merge(train, user, on=['userID'], how='left')
data = pd.merge(data, ad, on=['creativeID'], how='left')
data = pd.merge(data, position, on=['positionID'], how='left')
data = pd.merge(data, app_categories, on=['appID'], how='left')

print 'Data Processing Finished'



# <3> File Export  ********************************************
print 'Exporting File'

data.to_csv(conf['base_path'] + '/merge1.csv', encoding='gbk')

print 'Exporting File Finished'