1.查看dataframe占用空间
data = pd.read_csv('total_data.csv', index_col=0)
Int64Index: 91427 entries, 6095 to 91426
Data columns (total 13 columns):
dtypes: float64(9), int64(2), object(2)
memory usage: 14.4 MB
2.时间的处理
data['date'] = pd.to_datetime(data['date'])
Int64Index: 91427 entries, 6095 to 91426
Data columns (total 13 columns):
dtypes: datetime64[ns](1), float64(9), int64(2), object(1)
memory usage: 12.0 MB
3.修改数字
假设,我们一开始就定义好浮点数列的数据类型为float16
data = pd.read_csv('total_data.csv', index_col=0, dtype={'open': 'float16', 'close': 'float16', 'high': 'float16', 'low': 'float16','volume': 'float32',
然后info一下,
Int64Index: 91427 entries, 6095 to 91426
Data columns (total 13 columns):
dtypes: datetime64[ns](1), float16(8), float32(1), int32(1), int64(1), object(1)
memory usage: 7.1 MB
现在,内存占用只有7.1M了,连原来的一半都不到。
4.catrgory类
data['group'] = data['group'].astype('category')
data['sec_id'] = data['sec_id'].astype('category')
然后,我们看一下效果:
Int64Index: 91427entries, 6095 to 91426
Data columns(total 13 columns):
dtypes:category(2), datetime64[ns](1), float16(8), float32(1), int32(1)
memory usage: 3.7MB
5.注意: