创建子表并快速测试唯一性的封装(自定义)函数

最新推荐文章于 2024-05-28 11:34:53 发布

弎见

最新推荐文章于 2024-05-28 11:34:53 发布

阅读量176

点赞数

分类专栏：模板文章标签： python 封装 pandas groupby agg

本文链接：https://blog.csdn.net/sanjianjixiang/article/details/104737947

版权

模板专栏收录该内容

9 篇文章 0 订阅

订阅专栏

从一个大数据中创建子表并快速测试唯一性的辅助函数

player_index = 'playerShort'
player_cols = ['birthday', 'height', 'weight', 'position', 'photoID', 'rater1', 'rater2']

def get_subgroup(dataframe, g_index, g_columns):
    g = dataframe.groupby(g_index).agg({col:'nunique' for col in g_columns})
    if g[g > 1].dropna().shape[0] != 0:
        print("Warning: you probably assumed this had all unique values but it doesn't.")
    return dataframe.groupby(g_index).agg({col:'max' for col in g_columns})

players = get_subgroup(df, player_index, player_cols)
players.head()

在这里插入图片描述
保存数据, 并检测是否一致:

def save_subgroup(dataframe, g_index, subgroup_name, prefix = 'raw_'):
    save_subgroup_filename = ''.join([prefix, subgroup_name, '.csv.gz'])
    dataframe.to_csv(save_subgroup_filename, compression='gzip', encoding = 'UTF-8')
    test_df = pd.read_csv(save_subgroup_filename, compression='gzip', index_col = g_index, encoding='UTF-8')
    if dataframe.equals(test_df):
        print('Test-passed: we recover the equivalent subgroup dataframe.')
    else:
        print('Warning -- equivalence test!!! Double-check.')
        
save_subgroup(players, player_index, 'players')