python节省内存技巧_python中Pandas做处理时内存节省的技巧

1.查看dataframe占用空间

data = pd.read_csv('total_data.csv', index_col=0)

Int64Index: 91427 entries, 6095 to 91426

Data columns (total 13 columns):

dtypes: float64(9), int64(2), object(2)

memory usage: 14.4 MB

2.时间的处理

data['date'] = pd.to_datetime(data['date'])

Int64Index: 91427 entries, 6095 to 91426

Data columns (total 13 columns):

dtypes: datetime64[ns](1), float64(9), int64(2), object(1)

memory usage: 12.0 MB

3.修改数字

假设，我们一开始就定义好浮点数列的数据类型为float16

data = pd.read_csv('total_data.csv', index_col=0, dtype={'open': 'float16', 'close': 'float16', 'high': 'float16', 'low': 'float16','volume': 'float32',

然后info一下，

Int64Index: 91427 entries, 6095 to 91426

Data columns (total 13 columns):

dtypes: datetime64[ns](1), float16(8), float32(1), int32(1), int64(1), object(1)

memory usage: 7.1 MB

现在，内存占用只有7.1M了，连原来的一半都不到。

4.catrgory类

data['group'] = data['group'].astype('category')

data['sec_id'] = data['sec_id'].astype('category')

然后，我们看一下效果：

Int64Index: 91427entries, 6095 to 91426

Data columns(total 13 columns):

dtypes:category(2), datetime64[ns](1), float16(8), float32(1), int32(1)

memory usage: 3.7MB

5.注意：