机器学习—数据科学包(二)pandas
环境:
ipython
ipython是一个增强的Python交互解释器,它提供了一下功能:
- 动态对象自身:使用?或者??获取对象的文档注释,函数的原型定义,源代码等等。
- 支持使用通配符*进行模块搜索
- 支持代码自动补全
- 支持历史命令
- 其他魔法命令
cmd输入 ipython notebook 可自动打开网页版本
魔法函数
- 基于行的行魔法函数,以%开头,它使用当前行的剩余部分作为参数,参数不需要使用括号括起来。
- 基于单元格的单元格魔法函数, 以%%开头,使用当前行剩余部分以及后面的行作为参数。
%timeit的实例:
内置的魔法函数包括如下几种:
- 作用于代码的函数:%run,%edit,%save,%macro,%recall等
- 作用于shell的函数:%colors,%xmode,%autoindent,%automagic等。
- 其它函数,比如:%reset,%timeit,%%writefile,%load,%paste等。
如果当前命令只有一行,行魔法函数也可以省略%,但是单元格模范函数不能省略%%。
使用%magic获取获取IPython中魔法函数的详细介绍,如果需要获取某个魔法函数的详细信息,可以使用%somemagic?命令,此外,还可以使用%lsmagic获取所有的魔法函数。
ipython网页版本
IPython notebook目前已经成为用Python做教学、计算、科研的一个重要工具。IPython Notebook使用浏览器作为界面,向后台的IPython服务器发送请求,并显示结果。打开方式很简单,直接cmd进入到你要创建的目录下,然后输入命令:ipython notebook(现在升级了,打开主页显示的是jupyter,所以也可以使用jupyter notebook命令)。
pandas 入门:
官方文档:十分钟快速入门pandas
https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html
习惯上,我们会按下面格式引入所需要的包:
In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: import matplotlib.pyplot as plt
一、创建对象
1、可以通过传递一个list对象来创建一个Series,pandas会默认创建整型索引:
In[5]: s=pd.Series([1,3,5,np.nan,6,8])
In[6]: s
Out[6]:
0 1.0
1 3.0
2 5.0
3 NaN
4 6.0
5 8.0
dtype: float64
2、通过传递一个numpy array,时间索引以及列标签来创建一个DataFrame:
In[5]: dates=pd.date_range('20140201',periods=6)
In[6]: dates
Out[6]:
DatetimeIndex(['2014-02-01', '2014-02-02', '2014-02-03', '2014-02-04',
'2014-02-05', '2014-02-06'],
dtype='datetime64[ns]', freq='D')
In[7]: data=pd.DataFrame(np.random.randn(6,4),index=dates,columns=list('ABCD'))
In[8]: data
Out[8]:
A B C D
2014-02-01 -1.511038 -0.034441 -0.619328 0.921036
2014-02-02 0.936087 0.086909 -0.261515 0.627252
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594
2014-02-04 -1.609215 0.305436 0.695883 -1.213981
2014-02-05 0.159289 -0.748557 0.731476 1.809500
2014-02-06 0.966737 0.144001 -0.627874 0.251945
3、通过传递一个能够被转换成类似序列结构的字典对象来创建一个DataFrame:
df=pd.DataFrame(d)
df
Out[13]:
A B C D
0 1 2014-02-01 0 0
1 1 2014-02-01 1 1
2 1 2014-02-01 2 2
3 1 2014-02-01 3 3
4、查看不同列的数据类型:
df.dtypes
Out[14]:
A int64
B datetime64[ns]
C int64
D int32
dtype: object
df.A
Out[15]:
0 1
1 1
2 1
3 1
Name: A, dtype: int64
二、查看数据
1、 查看frame中头部和尾部的行:
data.head(3)
Out[18]:
A B C D
2014-02-01 -1.511038 -0.034441 -0.619328 0.921036
2014-02-02 0.936087 0.086909 -0.261515 0.627252
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594
data.tail(5)
Out[19]:
A B C D
2014-02-02 0.936087 0.086909 -0.261515 0.627252
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594
2014-02-04 -1.609215 0.305436 0.695883 -1.213981
2014-02-05 0.159289 -0.748557 0.731476 1.809500
2014-02-06 0.966737 0.144001 -0.627874 0.251945
2、 显示索引、列和底层的numpy数据:
data.index
Out[20]:
DatetimeIndex(['2014-02-01', '2014-02-02', '2014-02-03', '2014-02-04',
'2014-02-05', '2014-02-06'],
dtype='datetime64[ns]', freq='D')
data.columns
Out[21]: Index(['A', 'B', 'C', 'D'], dtype='object')
data.values
Out[22]:
array([[-1.51103778, -0.03444055, -0.61932786, 0.9210357 ],
[ 0.93608681, 0.08690872, -0.26151455, 0.62725179],
[-2.01032162, -1.65817512, -1.21468687, 0.4195938 ],
[-1.60921478, 0.30543596, 0.69588291, -1.2139811 ],
[ 0.15928935, -0.7485566 , 0.73147617, 1.80950001],
[ 0.96673722, 0.14400142, -0.62787421, 0.25194529]])
3、 describe()函数对于数据的快速统计汇总:
data.describe()
Out[23]:
A B C D
count 6.000000 6.000000 6.000000 6.000000
mean -0.511410 -0.317471 -0.216007 0.469224
std 1.355077 0.752053 0.782432 0.990787
min -2.010322 -1.658175 -1.214687 -1.213981
25% -1.584671 -0.570028 -0.625738 0.293857
50% -0.675874 0.026234 -0.440421 0.523423
75% 0.741887 0.129728 0.456534 0.847590
max 0.966737 0.305436 0.731476 1.809500
4、 对数据的转置:
In[24]: data.T
Out[24]:
2014-02-01 2014-02-02 ... 2014-02-05 2014-02-06
A -1.511038 0.936087 ... 0.159289 0.966737
B -0.034441 0.086909 ... -0.748557 0.144001
C -0.619328 -0.261515 ... 0.731476 -0.627874
D 0.921036 0.627252 ... 1.809500 0.251945
[4 rows x 6 columns]
5、 按轴进行排序
In[25]: data.sort_index(axis=1)
Out[25]:
A B C D
2014-02-01 -1.511038 -0.034441 -0.619328 0.921036
2014-02-02 0.936087 0.086909 -0.261515 0.627252
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594
2014-02-04 -1.609215 0.305436 0.695883 -1.213981
2014-02-05 0.159289 -0.748557 0.731476 1.809500
2014-02-06 0.966737 0.144001 -0.627874 0.251945
In[26]: data.sort_index(axis=1,ascending=False)
Out[26]:
D C B A
2014-02-01 0.921036 -0.619328 -0.034441 -1.511038
2014-02-02 0.627252 -0.261515 0.086909 0.936087
2014-02-03 0.419594 -1.214687 -1.658175 -2.010322
2014-02-04 -1.213981 0.695883 0.305436 -1.609215
2014-02-05 1.809500 0.731476 -0.748557 0.159289
2014-02-06 0.251945 -0.627874 0.144001 0.966737
6、 按值进行排序
In[27]: data.sort_values(by='C')
Out[27]:
A B C D
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594
2014-02-06 0.966737 0.144001 -0.627874 0.251945
2014-02-01 -1.511038 -0.034441 -0.619328 0.921036
2014-02-02 0.936087 0.086909 -0.261515 0.627252
2014-02-04 -1.609215 0.305436 0.695883 -1.213981
2014-02-05 0.159289 -0.748557 0.731476 1.809500
三、选择
- 获取
1、 选择一个单独的列,这将会返回一个Series,等同于data.A:
In[29]: data.A
Out[29]:
2014-02-01 -1.511038
2014-02-02 0.936087
2014-02-03 -2.010322
2014-02-04 -1.609215
2014-02-05 0.159289
2014-02-06 0.966737
Freq: D, Name: A, dtype: float64
2、 通过[]进行选择,这将会对行进行切片
In[30]: data[2:5]
Out[30]:
A B C D
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594
2014-02-04 -1.609215 0.305436 0.695883 -1.213981
2014-02-05 0.159289 -0.748557 0.731476 1.809500
- 通过标签选择
1、 通过标签来在多个轴上进行选择
In[32]: data.loc['2014-02-03':'2014-02-05']
Out[32]:
A B C D
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594
2014-02-04 -1.609215 0.305436 0.695883 -1.213981
2014-02-05 0.159289 -0.748557 0.731476 1.809500
2、 标签切片
In[33]: data.loc['2014-02-03':'2014-02-05','A':'D']
Out[33]:
A B C D
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594
2014-02-04 -1.609215 0.305436 0.695883 -1.213981
2014-02-05 0.159289 -0.748557 0.731476 1.809500
- 通过位置选择
1、 通过传递数值进行位置选择(选择的是行)
In[34]: data.iloc[3]
Out[34]:
A -1.609215
B 0.305436
C 0.695883
D -1.213981
Name: 2014-02-04 00:00:00, dtype: float64
2、 通过数值进行切片,与numpy/python中的情况类似
In[35]: data.iloc[2:3,1:2]
Out[35]:
B
2014-02-03 -1.658175
3、 通过指定一个位置的列表,与numpy/python中的情况类似
In [34]: df.iloc[[1,2,4],[0,2]]
Out[34]:
A C
2013-01-02 1.212112 0.119209
2013-01-03 -0.861849 -0.494929
2013-01-05 -0.424972 0.276232
- 通过布尔索引
1、 使用一个单独列的值来选择数据:
In [39]: df[df.A > 0]
Out[39]:
A B C D
2013-01-01 0.469112 -0.282863 -1.509059 -1.135632
2013-01-02 1.212112 -0.173215 0.119209 -1.044236
2013-01-04 0.721555 -0.706771 -1.039575 0.271860
2、 使用isin()方法来过滤:
In[39]: data2
Out[39]:
A B C D
2014-02-01 -1.511038 -0.034441 -0.619328 0.921036
2014-02-02 0.936087 0.086909 -0.261515 0.627252
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594
2014-02-04 -1.609215 0.305436 0.695883 -1.213981
2014-02-05 0.159289 -0.748557 0.731476 1.809500
2014-02-06 0.966737 0.144001 -0.627874 0.251945
In[40]: data2['E']=['a']*2+['b']*2+['c']*2
In[41]: data2
Out[41]:
A B C D E
2014-02-01 -1.511038 -0.034441 -0.619328 0.921036 a
2014-02-02 0.936087 0.086909 -0.261515 0.627252 a
2014-02-03 -2.010322 -1.658175 -1.214687 0.419594 b
2014-02-04 -1.609215 0.305436 0.695883 -1.213981 b
2014-02-05 0.159289 -0.748557 0.731476 1.809500 c
2014-02-06 0.966737 0.144001 -0.627874 0.251945 c
In[42]: data2[data2.E.isin(['a','c'])]
Out[42]:
A B C D E
2014-02-01 -1.511038 -0.034441 -0.619328 0.921036 a
2014-02-02 0.936087 0.086909 -0.261515 0.627252 a
2014-02-05 0.159289 -0.748557 0.731476 1.809500 c
2014-02-06 0.966737 0.144001 -0.627874 0.251945 c
- 设置
1、 设置一个新的列:
In [45]: s1 = pd.Series([1,2,3,4,5,6], index=pd.date_range('20130102', periods=6))
In [46]: s1
Out[46]:
2013-01-02 1
2013-01-03 2
2013-01-04 3
2013-01-05 4
2013-01-06 5
2013-01-07 6
Freq: D, dtype: int64
In [47]: df['F'] = s1
2、 通过标签设置新的值:
In [48]: df.at[dates[0],'A'] = 0
3、 通过位置设置新的值:
In [49]: df.iat[0,1] = 0
4、 通过一个numpy数组设置一组新值:
In [50]: df.loc[:,'D'] = np.array([5] * len(df))
上述操作结果如下:
In [51]: df
Out[51]:
A B C D F
2013-01-01 0.000000 0.000000 -1.509059 5 NaN
2013-01-02 1.212112 -0.173215 0.119209 5 1.0
2013-01-03 -0.861849 -2.104569 -0.494929 5 2.0
2013-01-04 0.721555 -0.706771 -1.039575 5 3.0
2013-01-05 -0.424972 0.567020 0.276232 5 4.0
2013-01-06 -0.673690 0.113648 -1.478427 5 5.0
5、 通过where操作来设置新的值:
In [52]: df2 = df.copy()
In [53]: df2[df2 > 0] = -df2
In [54]: df2
Out[54]:
A B C D F
2013-01-01 0.000000 0.000000 -1.509059 -5 NaN
2013-01-02 -1.212112 -0.173215 -0.119209 -5 -1.0
2013-01-03 -0.861849 -2.104569 -0.494929 -5 -2.0
2013-01-04 -0.721555 -0.706771 -1.039575 -5 -3.0
2013-01-05 -0.424972 -0.567020 -0.276232 -5 -4.0
2013-01-06 -0.673690 -0.113648 -1.478427 -5 -5.
四、缺失值处理
创建:
In[2]: import pandas as pd
In[3]: import numpy as np
In[4]: import matplotlib.pyplot as plt
In[5]: dates=pd.date_range('20160301',periods=6)
In[6]: df=pd.DataFrame(np.random.randn(6,4),index=dates,columns=list('ABCD'))
In[7]: df1=df.reindex(index=dates[0:4],columns=list(df.columns)+['E'])
In[8]: df1
Out[8]:
A B C D E
2016-03-01 0.899779 -0.347769 -0.664284 -0.646859 NaN
2016-03-02 -0.019930 -0.679697 0.207176 -0.186060 NaN
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958 NaN
2016-03-04 -0.101724 0.160194 -0.802420 1.033333 NaN
1、 reindex()方法可以对指定轴上的索引进行改变/增加/删除操作,这将返回原始数据的一个拷贝:
df1=df.reindex(index=dates[0:4],columns=list(df.columns)+['E'])
df1.loc[dates[1:3],'E']=2
A B C D E
2016-03-01 0.141177 1.391996 0.732199 0.336293 NaN
2016-03-02 -0.661433 -0.659423 1.146476 -0.602315 2.0
2016-03-03 0.804278 -0.429059 -1.477499 0.514814 2.0
2016-03-04 1.649037 -0.071460 1.911577 -0.173560 NaN
2、 去掉包含缺失值的行:
In[9]: df1.loc[dates[1:3],'E']=2
In[10]: df1
Out[10]:
A B C D E
2016-03-01 0.899779 -0.347769 -0.664284 -0.646859 NaN
2016-03-02 -0.019930 -0.679697 0.207176 -0.186060 2.0
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958 2.0
2016-03-04 -0.101724 0.160194 -0.802420 1.033333 NaN
In[11]: df1.dropna()
Out[11]:
A B C D E
2016-03-02 -0.019930 -0.679697 0.207176 -0.186060 2.0
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958 2.0
3、 对缺失值进行填充:
In[13]: df1.fillna(value=3)
Out[13]:
A B C D E
2016-03-01 0.899779 -0.347769 -0.664284 -0.646859 3.0
2016-03-02 -0.019930 -0.679697 0.207176 -0.186060 2.0
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958 2.0
2016-03-04 -0.101724 0.160194 -0.802420 1.033333 3.0
4、 对数据进行布尔填充:
In[15]: pd.isnull(df1)
Out[15]:
A B C D E
2016-03-01 False False False False True
2016-03-02 False False False False False
2016-03-03 False False False False False
2016-03-04 False False False False True
5、判断是否有空数据
pd.isnull(df1).any().any()
Out[16]: True
五、相关操作
- 统计
1、 执行描述性统计:
In [17]: df.mean()
Out[17]:
A -0.004474
B -0.383981
C -0.687758
D 5.000000
F 3.000000
dtype: float64
2、 对于拥有不同维度,需要对齐的对象进行操作。Pandas会自动的沿着指定的维度进行广播:
In[18]: s=pd.Series([1,3,5,np.nan,6,8],index=dates).shift(2)
In[19]: s
Out[19]:
2016-03-01 NaN
2016-03-02 NaN
2016-03-03 1.0
2016-03-04 3.0
2016-03-05 5.0
2016-03-06 NaN
Freq: D, dtype: float64
In[20]: df
Out[20]:
A B C D
2016-03-01 0.899779 -0.347769 -0.664284 -0.646859
2016-03-02 -0.019930 -0.679697 0.207176 -0.186060
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958
2016-03-04 -0.101724 0.160194 -0.802420 1.033333
2016-03-05 -0.955321 -0.539865 -1.144294 0.114223
2016-03-06 0.798588 -1.888797 -0.053185 1.463853
In[21]: df.sub(s,axis='index')
Out[21]:
A B C D
2016-03-01 NaN NaN NaN NaN
2016-03-02 NaN NaN NaN NaN
2016-03-03 -1.007573 -1.521074 -1.409476 -1.286958
2016-03-04 -3.101724 -2.839806 -3.802420 -1.966667
2016-03-05 -5.955321 -5.539865 -6.144294 -4.885777
2016-03-06 NaN NaN NaN NaN
- apply
1、 对数据应用函数:
/***累加****/
In[22]: df.apply(np.cumsum)
Out[22]:
A B C D
2016-03-01 0.899779 -0.347769 -0.664284 -0.646859
2016-03-02 0.879850 -1.027466 -0.457108 -0.832919
2016-03-03 0.872277 -1.548540 -0.866584 -1.119876
2016-03-04 0.770553 -1.388346 -1.669004 -0.086543
2016-03-05 -0.184767 -1.928211 -2.813298 0.027679
2016-03-06 0.613820 -3.817008 -2.866482 1.491532
/***最大值减最小值****/
In[23]: df.apply(lambda x:x.max()-x.min())
Out[23]:
A 1.855100
B 2.048991
C 1.351470
D 2.110712
dtype: float64
- 次数显示
1、记录次数
In[5]: s=pd.Series(np.random.randint(0,7,size=10))
In[6]: s
Out[6]:
0 6
1 4
2 0
3 2
4 4
5 0
6 6
7 4
8 1
9 4
dtype: int32
In[7]: s.value_counts()
Out[7]:
4 4
6 2
0 2
2 1
1 1
dtype: int64
2、显示最多次数
s.mode()
Out[9]:
0 4
dtype: int32
六、合并
Pandas提供了大量的方法能够轻松的对Series,DataFrame和Panel对象进行各种符合各种逻辑关系的合并操作。
- pd .concat
In[12]: df
Out[12]:
A B C D
0 -0.023904 0.370157 -0.189051 -1.228684
1 -0.209592 -0.348743 -1.136188 -1.103672
2 -1.014317 -1.806632 1.917631 0.450594
3 -1.035946 -0.359108 -0.570582 0.153201
4 1.667096 1.474152 1.624510 -1.513115
5 -1.091161 0.957389 0.447147 -2.581356
6 0.963045 0.575139 1.438793 -1.556901
7 -0.437601 0.451941 -0.137079 -0.050907
8 -0.629690 -0.556138 0.469743 -1.837013
9 -0.368121 -1.351038 -0.355778 -0.794521
In[13]: df.iloc[:3]
Out[13]:
A B C D
0 -0.023904 0.370157 -0.189051 -1.228684
1 -0.209592 -0.348743 -1.136188 -1.103672
2 -1.014317 -1.806632 1.917631 0.450594
In[14]: df.iloc[3:7]
Out[14]:
A B C D
3 -1.035946 -0.359108 -0.570582 0.153201
4 1.667096 1.474152 1.624510 -1.513115
5 -1.091161 0.957389 0.447147 -2.581356
6 0.963045 0.575139 1.438793 -1.556901
In[15]: df.iloc[7:]
Out[15]:
A B C D
7 -0.437601 0.451941 -0.137079 -0.050907
8 -0.629690 -0.556138 0.469743 -1.837013
9 -0.368121 -1.351038 -0.355778 -0.794521
In[16]: df1=pd.concat([df.iloc[:3],df.iloc[3:7],df.iloc[7:]])
In[17]:
In[17]: df1
Out[17]:
A B C D
0 -0.023904 0.370157 -0.189051 -1.228684
1 -0.209592 -0.348743 -1.136188 -1.103672
2 -1.014317 -1.806632 1.917631 0.450594
3 -1.035946 -0.359108 -0.570582 0.153201
4 1.667096 1.474152 1.624510 -1.513115
5 -1.091161 0.957389 0.447147 -2.581356
6 0.963045 0.575139 1.438793 -1.556901
7 -0.437601 0.451941 -0.137079 -0.050907
8 -0.629690 -0.556138 0.469743 -1.837013
9 -0.368121 -1.351038 -0.355778 -0.794521
- pd .merge
In[18]: left=pd.DataFrame({'key':['foo','foo'],'lval':[1,2]})
In[19]: right=pd.DataFrame({'key':['foo','foo'],'rval':[4,5]})
In[20]: left
Out[20]:
key lval
0 foo 1
1 foo 2
In[21]: right
Out[21]:
key rval
0 foo 4
1 foo 5
In[22]: pd.merge(left,right,on='key')
Out[22]:
key lval rval
0 foo 1 4
1 foo 1 5
2 foo 2 4
3 foo 2 5
- Append 将一行连接到一个DataFrame上
In [82]: df = pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'])
In [83]: df
Out[83]:
A B C D
0 1.346061 1.511763 1.627081 -0.990582
1 -0.441652 1.211526 0.268520 0.024580
2 -1.577585 0.396823 -0.105381 -0.532532
3 1.453749 1.208843 -0.080952 -0.264610
4 -0.727965 -0.589346 0.339969 -0.693205
5 -0.339355 0.593616 0.884345 1.591431
6 0.141809 0.220390 0.435589 0.192451
7 -0.096701 0.803351 1.715071 -0.708758
In [84]: s = df.iloc[3]
In [85]: df.append(s, ignore_index=True)
Out[85]:
A B C D
0 1.346061 1.511763 1.627081 -0.990582
1 -0.441652 1.211526 0.268520 0.024580
2 -1.577585 0.396823 -0.105381 -0.532532
3 1.453749 1.208843 -0.080952 -0.264610
4 -0.727965 -0.589346 0.339969 -0.693205
5 -0.339355 0.593616 0.884345 1.591431
6 0.141809 0.220390 0.435589 0.192451
7 -0.096701 0.803351 1.715071 -0.708758
8 1.453749 1.208843 -0.080952 -0.264610
七、分组
按照一定的规则分组,并进行操作:
- 单索引
In[24]: df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
...: 'foo', 'bar', 'foo', 'foo'],
...: 'B' : ['one', 'one', 'two', 'three',
...: 'two', 'two', 'one', 'three'],
...: 'C' : np.random.randn(8),
...: 'D' : np.random.randn(8)})
In[25]: df
Out[25]:
A B C D
0 foo one -0.133907 1.086158
1 bar one -0.248065 -0.543809
2 foo two -0.185406 0.856915
3 bar three 0.732933 -0.727144
4 foo two -1.222730 0.649772
5 bar two -0.705964 0.852976
6 foo one -1.357128 -0.236820
7 foo three -0.455601 -0.163037
In[26]: df.groupby('A').sum()
Out[26]:
C D
A
bar -0.221096 -0.417976
foo -3.354772 2.192987
- 双索引
In[28]: df.groupby(['A','B']).sum()
Out[28]:
C D
A B
bar one -0.248065 -0.543809
three 0.732933 -0.727144
two -0.705964 0.852976
foo one -1.491034 0.849337
three -0.455601 -0.163037
two -1.408136 1.506687
八、重塑
- ** stack**
tuples=list(zip(*[['bar','bar','baz','baz','foo','foo','qux','qux'],
['one','two','one','two','one','two','one','two']]))
index=pd.MultiIndex.from_tuples(tuples,names=['first','second'])
df=pd.DataFrame(np.random.randn(8,2),index=index,columns=['A','B'])
stacked=df.stack()
Out[12]:
first second
bar one A 0.204809
B -0.088132
two A -0.499254
B -0.463513
baz one A -0.149506
B 1.153880
two A 2.227171
B -1.031684
foo one A 0.038368
B -0.763699
two A -0.304946
B 0.198953
qux one A -0.535454
B -0.528770
two A -1.305537
B -0.529129
dtype: float64
- ** stack()的逆运算**
stacked.unstack().unstack()
A B
second one two one two
first
bar 0.204809 -0.499254 -0.088132 -0.463513
baz -0.149506 2.227171 1.153880 -1.031684
foo 0.038368 -0.304946 -0.763699 0.198953
qux -0.535454 -1.305537 -0.528770 -0.529129
- 数据透视
df = pd.DataFrame({'A' : ['one', 'one', 'two', 'three'] * 3,
'B' : ['A', 'B', 'C'] * 4,
'C' : ['foo', 'foo', 'foo', 'bar', 'bar', 'bar'] * 2,
'D' : np.random.randn(12),
'E' : np.random.randn(12)})
A B C D E
0 one A foo 0.260388 -1.040629
1 one B foo 1.204708 0.133854
2 two C foo -1.000894 -0.522464
3 three A bar -1.294604 0.434995
4 one B bar 0.226309 0.560666
5 one C bar -0.572290 1.359220
6 two A foo 0.637389 -0.025343
7 three B foo 0.530362 -0.739961
8 one C foo -0.463693 1.364281
9 one A bar -0.134875 -1.148102
10 two B bar -1.529663 0.062310
11 three C bar 1.414118 0.805989
df.pivot_table(values=['D'],index=['A','B'],columns=['C'])
D
C bar foo
A B
one A -0.134875 0.260388
B 0.226309 1.204708
C -0.572290 -0.463693
three A -1.294604 NaN
B NaN 0.530362
C 1.414118 NaN
two A NaN 0.637389
B -1.529663 NaN
C NaN -1.000894
当数据透视表对应多个值时求取得是平均值,对应空值时NAN
九、时间序列
Pandas在对频率转换进行重新采样时拥有简单、强大且高效的功能(如将按秒采样的数据转换为按2分钟为单位进行采样的数据)。这种操作在金融领域非常常见。
- 时间序列的创建
/****以2min为一个单位*****/
rng=pd.date_range('20160301',periods=600,freq='s')
DatetimeIndex(['2016-03-01 00:00:00', '2016-03-01 00:00:01',
'2016-03-01 00:00:02', '2016-03-01 00:00:03',
'2016-03-01 00:00:04', '2016-03-01 00:00:05',
'2016-03-01 00:00:06', '2016-03-01 00:00:07',
'2016-03-01 00:00:08', '2016-03-01 00:00:09',
...
'2016-03-01 00:09:50', '2016-03-01 00:09:51',
'2016-03-01 00:09:52', '2016-03-01 00:09:53',
'2016-03-01 00:09:54', '2016-03-01 00:09:55',
'2016-03-01 00:09:56', '2016-03-01 00:09:57',
'2016-03-01 00:09:58', '2016-03-01 00:09:59'],
dtype='datetime64[ns]', length=600, freq='S')
s=pd.Series(np.random.randint(0,500,len(rng)),index=rng)
s.resample('2Min',how='sum')
2016-03-01 00:00:00 30109
2016-03-01 00:02:00 32759
2016-03-01 00:04:00 32095
2016-03-01 00:06:00 30774
2016-03-01 00:08:00 29775
Freq: 2T, dtype: int32
In[29]: rng=pd.period_range('2000Q1','2016Q1',freq='Q')
In[30]: rng
Out[30]:
PeriodIndex(['2000Q1', '2000Q2', '2000Q3', '2000Q4', '2001Q1', '2001Q2',
'2001Q3', '2001Q4', '2002Q1', '2002Q2', '2002Q3', '2002Q4',
'2003Q1', '2003Q2', '2003Q3', '2003Q4', '2004Q1', '2004Q2',
'2004Q3', '2004Q4', '2005Q1', '2005Q2', '2005Q3', '2005Q4',
'2006Q1', '2006Q2', '2006Q3', '2006Q4', '2007Q1', '2007Q2',
'2007Q3', '2007Q4', '2008Q1', '2008Q2', '2008Q3', '2008Q4',
'2009Q1', '2009Q2', '2009Q3', '2009Q4', '2010Q1', '2010Q2',
'2010Q3', '2010Q4', '2011Q1', '2011Q2', '2011Q3', '2011Q4',
'2012Q1', '2012Q2', '2012Q3', '2012Q4', '2013Q1', '2013Q2',
'2013Q3', '2013Q4', '2014Q1', '2014Q2', '2014Q3', '2014Q4',
'2015Q1', '2015Q2', '2015Q3', '2015Q4', '2016Q1'],
dtype='period[Q-DEC]', freq='Q-DEC')
转化为时间日期格式:
In[31]: rng.to_timestamp()
Out[31]:
DatetimeIndex(['2000-01-01', '2000-04-01', '2000-07-01', '2000-10-01',
'2001-01-01', '2001-04-01', '2001-07-01', '2001-10-01',
'2002-01-01', '2002-04-01', '2002-07-01', '2002-10-01',
'2003-01-01', '2003-04-01', '2003-07-01', '2003-10-01',
'2004-01-01', '2004-04-01', '2004-07-01', '2004-10-01',
'2005-01-01', '2005-04-01', '2005-07-01', '2005-10-01',
'2006-01-01', '2006-04-01', '2006-07-01', '2006-10-01',
'2007-01-01', '2007-04-01', '2007-07-01', '2007-10-01',
'2008-01-01', '2008-04-01', '2008-07-01', '2008-10-01',
'2009-01-01', '2009-04-01', '2009-07-01', '2009-10-01',
'2010-01-01', '2010-04-01', '2010-07-01', '2010-10-01',
'2011-01-01', '2011-04-01', '2011-07-01', '2011-10-01',
'2012-01-01', '2012-04-01', '2012-07-01', '2012-10-01',
'2013-01-01', '2013-04-01', '2013-07-01', '2013-10-01',
'2014-01-01', '2014-04-01', '2014-07-01', '2014-10-01',
'2015-01-01', '2015-04-01', '2015-07-01', '2015-10-01',
'2016-01-01'],
dtype='datetime64[ns]', freq='QS-OCT')
- 日期相减
In[32]: pd.Timestamp('20160301')-pd.Timestamp('20160201')
Out[32]: Timedelta('29 days 00:00:00')
- 日期相加
十、Categorical
1、 创建Categorical数据结构
In[33]: df=pd.DataFrame({'id':[1,2,3,4,5],'row_grade':['a','b','a','d','d']})
In[34]: df['grade']=df['row_grade'].astype('category')
In[35]: df
Out[35]:
id row_grade grade
0 1 a a
1 2 b b
2 3 a a
3 4 d d
4 5 d d
2、 将Categorical类型数据重命名为更有意义的名称:
df['grade'].cat.categories=['very good','good','bad']
df
Out[37]:
id row_grade grade
0 1 a very good
1 2 b good
2 3 a very good
3 4 d bad
4 5 d bad
3、 排序是按照Categorical的顺序进行的而不是按照字典顺序进行:
In[38]: df.sort_values(by='grade')
Out[38]:
id row_grade grade
0 1 a very good
2 3 a very good
1 2 b good
3 4 d bad
4 5 d bad
十一、画图
t=pd.Series(np.random.randn(1000),index=pd.date_range(('20000101'),periods=1000))
t=t.cumsum()
t.plot()
plt.show()
对于DataFrame来说,plot是一种将所有列及其标签进行绘制的简便方法:
In [40]: df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,
.....: columns=['A', 'B', 'C', 'D'])
.....:
In [41]: df = df.cumsum()
In [42]: plt.figure(); df.plot(); plt.legend(loc='best')
Out[42]: <matplotlib.legend.Legend at 0x112854d90
十二、导入和保存数据
- CSV
1、写入CSV文件并读出
In[49]: df=pd.DataFrame(np.random.randn(100,4))
In[50]: df=pd.DataFrame(np.random.randn(100,4),columns=list('ABCD'))
In[51]: df
Out[51]:
A B C D
0 0.378867 -0.091620 1.090019 0.999421
1 2.251850 0.238909 -0.901346 0.015554
2 0.209695 -2.048660 -0.955254 0.344090
3 -0.130926 0.164686 1.340037 2.832207
4 0.166250 -0.101414 -0.232638 0.456991
5 0.304952 -0.403781 -0.130427 -0.956354
6 -0.429590 0.773878 -0.996998 -0.292732
7 -0.473208 0.674366 0.960866 0.682791
8 -1.840992 -0.526458 -0.420458 1.559731
9 -0.068624 0.315819 1.037631 -0.446265
10 0.261860 0.364727 0.343445 -0.109746
11 0.311507 -0.532801 -1.150832 0.302602
12 -0.841183 0.589915 -0.161598 0.378613
13 1.419394 0.097689 0.224020 -0.720346
14 0.468242 -0.635890 -0.413119 -0.160551
15 -1.375062 -0.560435 2.428874 -1.007550
16 -0.063632 1.631429 1.021518 -0.053831
17 -1.696037 -0.328426 0.440562 -0.590583
18 -0.661396 -0.909790 0.087819 0.889130
19 2.111948 -1.750826 1.747665 -1.459147
20 1.205531 -0.862482 0.062849 -0.019009
21 1.743419 -0.564083 1.311440 -0.529935
22 0.016245 -0.102664 -1.099839 -1.010344
23 -0.482878 0.826072 -0.163725 0.097180
24 -0.139670 0.902446 -0.460139 -0.549083
25 -1.438988 -0.205327 -0.344405 -1.374123
26 0.308278 -1.064430 0.329222 -0.577498
27 0.733006 0.026945 -0.092934 -0.341411
28 -1.464381 -1.634260 0.909248 -1.700494
29 0.039253 -1.598910 1.284848 0.402043
.. ... ... ... ...
70 -1.385963 1.633525 -0.422429 -0.886571
71 -1.471958 -0.001905 1.438166 0.150414
72 0.381426 0.403522 0.464480 0.191731
73 -0.280404 1.045549 -1.364438 -1.526277
74 0.458306 -0.996155 -1.699094 -1.194917
75 -0.307120 -2.056273 -1.043319 0.862359
76 1.864204 0.437932 -0.404072 -0.128899
77 1.271321 -0.747139 1.400846 -0.701221
78 1.589635 1.048788 0.706948 1.156733
79 1.378117 -0.179104 -2.180511 0.049153
80 0.555157 0.108048 -0.191770 -1.194258
81 -1.539549 -0.623444 2.785096 -0.408042
82 -1.062381 -0.007418 0.794200 0.129454
83 -0.177113 1.465207 0.538259 0.425892
84 0.053148 -1.966307 -0.165101 1.628191
85 -1.816071 -0.135639 0.147840 -0.383169
86 -0.328421 -0.063588 -0.039663 0.846303
87 -0.699656 -0.687535 0.586631 0.013061
88 0.074753 -0.858002 -0.647638 0.481088
89 -1.087874 0.883394 -0.654534 -1.213953
90 -1.460692 -0.038029 -0.706556 0.261179
91 0.759244 -1.255425 0.218774 0.058548
92 1.136994 -1.315591 -1.520542 -0.336128
93 -0.825148 -0.736311 -0.653980 0.506639
94 0.743375 0.457888 0.811990 1.193933
95 -0.055319 -0.188443 -1.926626 -0.186323
96 -0.277529 1.293196 0.026766 -1.764860
97 2.038703 0.785085 0.330018 0.056930
98 -0.532230 -1.306918 -0.666728 1.322078
99 -1.026237 1.315298 0.208255 -1.145776
[100 rows x 4 columns]
In[52]: df.to_csv('data.csv')
In[53]: pd.read_csv('data.csv',index_col=0)
Out[53]:
A B C D
0 0.378867 -0.091620 1.090019 0.999421
1 2.251850 0.238909 -0.901346 0.015554
2 0.209695 -2.048660 -0.955254 0.344090
3 -0.130926 0.164686 1.340037 2.832207
4 0.166250 -0.101414 -0.232638 0.456991
5 0.304952 -0.403781 -0.130427 -0.956354
6 -0.429590 0.773878 -0.996998 -0.292732
7 -0.473208 0.674366 0.960866 0.682791
8 -1.840992 -0.526458 -0.420458 1.559731
9 -0.068624 0.315819 1.037631 -0.446265
10 0.261860 0.364727 0.343445 -0.109746
11 0.311507 -0.532801 -1.150832 0.302602
12 -0.841183 0.589915 -0.161598 0.378613
13 1.419394 0.097689 0.224020 -0.720346
14 0.468242 -0.635890 -0.413119 -0.160551
15 -1.375062 -0.560435 2.428874 -1.007550
16 -0.063632 1.631429 1.021518 -0.053831
17 -1.696037 -0.328426 0.440562 -0.590583
18 -0.661396 -0.909790 0.087819 0.889130
19 2.111948 -1.750826 1.747665 -1.459147
20 1.205531 -0.862482 0.062849 -0.019009
21 1.743419 -0.564083 1.311440 -0.529935
22 0.016245 -0.102664 -1.099839 -1.010344
23 -0.482878 0.826072 -0.163725 0.097180
24 -0.139670 0.902446 -0.460139 -0.549083
25 -1.438988 -0.205327 -0.344405 -1.374123
26 0.308278 -1.064430 0.329222 -0.577498
27 0.733006 0.026945 -0.092934 -0.341411
28 -1.464381 -1.634260 0.909248 -1.700494
29 0.039253 -1.598910 1.284848 0.402043
.. ... ... ... ...
70 -1.385963 1.633525 -0.422429 -0.886571
71 -1.471958 -0.001905 1.438166 0.150414
72 0.381426 0.403522 0.464480 0.191731
73 -0.280404 1.045549 -1.364438 -1.526277
74 0.458306 -0.996155 -1.699094 -1.194917
75 -0.307120 -2.056273 -1.043319 0.862359
76 1.864204 0.437932 -0.404072 -0.128899
77 1.271321 -0.747139 1.400846 -0.701221
78 1.589635 1.048788 0.706948 1.156733
79 1.378117 -0.179104 -2.180511 0.049153
80 0.555157 0.108048 -0.191770 -1.194258
81 -1.539549 -0.623444 2.785096 -0.408042
82 -1.062381 -0.007418 0.794200 0.129454
83 -0.177113 1.465207 0.538259 0.425892
84 0.053148 -1.966307 -0.165101 1.628191
85 -1.816071 -0.135639 0.147840 -0.383169
86 -0.328421 -0.063588 -0.039663 0.846303
87 -0.699656 -0.687535 0.586631 0.013061
88 0.074753 -0.858002 -0.647638 0.481088
89 -1.087874 0.883394 -0.654534 -1.213953
90 -1.460692 -0.038029 -0.706556 0.261179
91 0.759244 -1.255425 0.218774 0.058548
92 1.136994 -1.315591 -1.520542 -0.336128
93 -0.825148 -0.736311 -0.653980 0.506639
94 0.743375 0.457888 0.811990 1.193933
95 -0.055319 -0.188443 -1.926626 -0.186323
96 -0.277529 1.293196 0.026766 -1.764860
97 2.038703 0.785085 0.330018 0.056930
98 -0.532230 -1.306918 -0.666728 1.322078
99 -1.026237 1.315298 0.208255 -1.145776
[100 rows x 4 columns]