28. Pandas的Categorical数据类型可以降低数据存储提升计算速度

Pandas的Categorical数据类型可以降低数据存储提升计算速度

1、读取数据

import pandas as pd

df = pd.read_csv("./datas/movielens-1m/users.dat",
                 sep="::",
                 engine="python",
                 header=None,
                 names="UserID::Gender::Age::Occupation::Zip-code".split("::"))

df.head()

	UserID	Gender	Age	Occupation	Zip-code
0	1	F	1	10	48067
1	2	M	56	16	70072
2	3	M	25	15	55117
3	4	M	45	7	02460
4	5	M	25	20	55455

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6040 entries, 0 to 6039
Data columns (total 5 columns):
UserID        6040 non-null int64
Gender        6040 non-null object
Age           6040 non-null int64
Occupation    6040 non-null int64
Zip-code      6040 non-null object
dtypes: int64(3), object(2)
memory usage: 236.1+ KB

df.info(memory_usage="deep")

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6040 entries, 0 to 6039
Data columns (total 5 columns):
UserID        6040 non-null int64
Gender        6040 non-null object
Age           6040 non-null int64
Occupation    6040 non-null int64
Zip-code      6040 non-null object
dtypes: int64(3), object(2)
memory usage: 873.4 KB

df_cat = df.copy()
df_cat.head()

	UserID	Gender	Age	Occupation	Zip-code
0	1	F	1	10	48067
1	2	M	56	16	70072
2	3	M	25	15	55117
3	4	M	45	7	02460
4	5	M	25	20	55455

2、使用categorical类型降低存储量

df_cat["Gender"] = df_cat["Gender"].astype("category")

df_cat.info(memory_usage="deep")

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6040 entries, 0 to 6039
Data columns (total 5 columns):
UserID        6040 non-null int64
Gender        6040 non-null category
Age           6040 non-null int64
Occupation    6040 non-null int64
Zip-code      6040 non-null object
dtypes: category(1), int64(3), object(1)
memory usage: 513.8 KB

df_cat.head()

	UserID	Gender	Age	Occupation	Zip-code
0	1	F	1	10	48067
1	2	M	56	16	70072
2	3	M	25	15	55117
3	4	M	45	7	02460
4	5	M	25	20	55455

df_cat["Gender"].value_counts()

M    4331
F    1709
Name: Gender, dtype: int64

3、提升运算速度

%timeit df.groupby("Gender").size()

564 µs ± 10.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit df_cat.groupby("Gender").size()

324 µs ± 5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)