机器学习—数据科学包（二）pandas快速入门

最新推荐文章于 2022-04-06 12:12:10 发布

�春风亲吻我像蛋蛋蛋挞。

最新推荐文章于 2022-04-06 12:12:10 发布

阅读量401

点赞数 1

本文链接：https://blog.csdn.net/dandandandanta/article/details/104281563

版权

机器学习—数据科学包（二）pandas

环境：

ipython

ipython是一个增强的Python交互解释器，它提供了一下功能：

动态对象自身：使用？或者？？获取对象的文档注释，函数的原型定义，源代码等等。
支持使用通配符*进行模块搜索
支持代码自动补全
支持历史命令
其他魔法命令

cmd输入 ipython notebook 可自动打开网页版本

魔法函数

基于行的行魔法函数，以%开头，它使用当前行的剩余部分作为参数，参数不需要使用括号括起来。
基于单元格的单元格魔法函数，以%%开头，使用当前行剩余部分以及后面的行作为参数。

%timeit的实例：
在这里插入图片描述
内置的魔法函数包括如下几种：

作用于代码的函数：%run，%edit，%save，%macro，%recall等
作用于shell的函数：%colors，%xmode，%autoindent，%automagic等。
其它函数，比如：%reset，%timeit，%%writefile，%load，%paste等。
如果当前命令只有一行，行魔法函数也可以省略%，但是单元格模范函数不能省略%%。

使用%magic获取获取IPython中魔法函数的详细介绍，如果需要获取某个魔法函数的详细信息，可以使用%somemagic?命令，此外，还可以使用%lsmagic获取所有的魔法函数。

ipython网页版本
IPython notebook目前已经成为用Python做教学、计算、科研的一个重要工具。IPython Notebook使用浏览器作为界面，向后台的IPython服务器发送请求，并显示结果。打开方式很简单，直接cmd进入到你要创建的目录下，然后输入命令：ipython notebook（现在升级了，打开主页显示的是jupyter，所以也可以使用jupyter notebook命令）。
在这里插入图片描述

pandas 入门：

官方文档：十分钟快速入门pandas
https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html

习惯上，我们会按下面格式引入所需要的包：

In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: import matplotlib.pyplot as plt

一、创建对象

1、可以通过传递一个list对象来创建一个Series，pandas会默认创建整型索引：

In[5]: s=pd.Series([1,3,5,np.nan,6,8])
In[6]: s
Out[6]: 
0    1.0
1    3.0
2    5.0
3    NaN
4    6.0
5    8.0
dtype: float64

2、通过传递一个numpy array，时间索引以及列标签来创建一个DataFrame：

In[5]: dates=pd.date_range('20140201',periods=6)
In[6]: dates
Out[6]: 
DatetimeIndex(['2014-02-01', '2014-02-02', '2014-02-03', '2014-02-04',
               '2014-02-05', '2014-02-06'],
              dtype='datetime64[ns]', freq='D')
In[7]: data=pd.DataFrame(np.random.randn(6,4),index=dates,columns=list('ABCD'))
In[8]: data
Out[8]: 
                   A         B         C         D
2014-02-01 -1.511038 -0.034441 -0.619328  0.921036
2014-02-02  0.936087  0.086909 -0.261515  0.627252
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594
2014-02-04 -1.609215  0.305436  0.695883 -1.213981
2014-02-05  0.159289 -0.748557  0.731476  1.809500
2014-02-06  0.966737  0.144001 -0.627874  0.251945

3、通过传递一个能够被转换成类似序列结构的字典对象来创建一个DataFrame：

df=pd.DataFrame(d)
df
Out[13]: 
   A          B  C  D
0  1 2014-02-01  0  0
1  1 2014-02-01  1  1
2  1 2014-02-01  2  2
3  1 2014-02-01  3  3

4、查看不同列的数据类型：

df.dtypes
Out[14]: 
A             int64
B    datetime64[ns]
C             int64
D             int32
dtype: object
df.A
Out[15]: 
0    1
1    1
2    1
3    1
Name: A, dtype: int64

二、查看数据

1、查看frame中头部和尾部的行：

data.head(3)
Out[18]: 
                   A         B         C         D
2014-02-01 -1.511038 -0.034441 -0.619328  0.921036
2014-02-02  0.936087  0.086909 -0.261515  0.627252
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594
data.tail(5)
Out[19]: 
                   A         B         C         D
2014-02-02  0.936087  0.086909 -0.261515  0.627252
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594
2014-02-04 -1.609215  0.305436  0.695883 -1.213981
2014-02-05  0.159289 -0.748557  0.731476  1.809500
2014-02-06  0.966737  0.144001 -0.627874  0.251945

2、显示索引、列和底层的numpy数据：

data.index
Out[20]: 
DatetimeIndex(['2014-02-01', '2014-02-02', '2014-02-03', '2014-02-04',
               '2014-02-05', '2014-02-06'],
              dtype='datetime64[ns]', freq='D')
data.columns
Out[21]: Index(['A', 'B', 'C', 'D'], dtype='object')
data.values
Out[22]: 
array([[-1.51103778, -0.03444055, -0.61932786,  0.9210357 ],
       [ 0.93608681,  0.08690872, -0.26151455,  0.62725179],
       [-2.01032162, -1.65817512, -1.21468687,  0.4195938 ],
       [-1.60921478,  0.30543596,  0.69588291, -1.2139811 ],
       [ 0.15928935, -0.7485566 ,  0.73147617,  1.80950001],
       [ 0.96673722,  0.14400142, -0.62787421,  0.25194529]])

3、 describe()函数对于数据的快速统计汇总：

data.describe()
Out[23]: 
              A         B         C         D
count  6.000000  6.000000  6.000000  6.000000
mean  -0.511410 -0.317471 -0.216007  0.469224
std    1.355077  0.752053  0.782432  0.990787
min   -2.010322 -1.658175 -1.214687 -1.213981
25%   -1.584671 -0.570028 -0.625738  0.293857
50%   -0.675874  0.026234 -0.440421  0.523423
75%    0.741887  0.129728  0.456534  0.847590
max    0.966737  0.305436  0.731476  1.809500

4、对数据的转置：

In[24]: data.T
Out[24]: 
   2014-02-01  2014-02-02     ...      2014-02-05  2014-02-06
A   -1.511038    0.936087     ...        0.159289    0.966737
B   -0.034441    0.086909     ...       -0.748557    0.144001
C   -0.619328   -0.261515     ...        0.731476   -0.627874
D    0.921036    0.627252     ...        1.809500    0.251945

[4 rows x 6 columns]

5、按轴进行排序

In[25]: data.sort_index(axis=1)
Out[25]: 
                   A         B         C         D
2014-02-01 -1.511038 -0.034441 -0.619328  0.921036
2014-02-02  0.936087  0.086909 -0.261515  0.627252
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594
2014-02-04 -1.609215  0.305436  0.695883 -1.213981
2014-02-05  0.159289 -0.748557  0.731476  1.809500
2014-02-06  0.966737  0.144001 -0.627874  0.251945
In[26]: data.sort_index(axis=1,ascending=False)
Out[26]: 
                   D         C         B         A
2014-02-01  0.921036 -0.619328 -0.034441 -1.511038
2014-02-02  0.627252 -0.261515  0.086909  0.936087
2014-02-03  0.419594 -1.214687 -1.658175 -2.010322
2014-02-04 -1.213981  0.695883  0.305436 -1.609215
2014-02-05  1.809500  0.731476 -0.748557  0.159289
2014-02-06  0.251945 -0.627874  0.144001  0.966737

6、按值进行排序

In[27]: data.sort_values(by='C')
Out[27]: 
                   A         B         C         D
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594
2014-02-06  0.966737  0.144001 -0.627874  0.251945
2014-02-01 -1.511038 -0.034441 -0.619328  0.921036
2014-02-02  0.936087  0.086909 -0.261515  0.627252
2014-02-04 -1.609215  0.305436  0.695883 -1.213981
2014-02-05  0.159289 -0.748557  0.731476  1.809500

三、选择

获取
1、选择一个单独的列，这将会返回一个Series，等同于data.A：

In[29]: data.A
Out[29]: 
2014-02-01   -1.511038
2014-02-02    0.936087
2014-02-03   -2.010322
2014-02-04   -1.609215
2014-02-05    0.159289
2014-02-06    0.966737
Freq: D, Name: A, dtype: float64

2、通过[]进行选择，这将会对行进行切片

In[30]: data[2:5]
Out[30]: 
                   A         B         C         D
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594
2014-02-04 -1.609215  0.305436  0.695883 -1.213981
2014-02-05  0.159289 -0.748557  0.731476  1.809500

通过标签选择

1、通过标签来在多个轴上进行选择

In[32]: data.loc['2014-02-03':'2014-02-05']
Out[32]: 
                   A         B         C         D
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594
2014-02-04 -1.609215  0.305436  0.695883 -1.213981
2014-02-05  0.159289 -0.748557  0.731476  1.809500

2、标签切片

In[33]: data.loc['2014-02-03':'2014-02-05','A':'D']
Out[33]: 
                   A         B         C         D
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594
2014-02-04 -1.609215  0.305436  0.695883 -1.213981
2014-02-05  0.159289 -0.748557  0.731476  1.809500

通过位置选择
1、通过传递数值进行位置选择（选择的是行）

In[34]: data.iloc[3]
Out[34]: 
A   -1.609215
B    0.305436
C    0.695883
D   -1.213981
Name: 2014-02-04 00:00:00, dtype: float64

2、通过数值进行切片，与numpy/python中的情况类似

In[35]: data.iloc[2:3,1:2]
Out[35]: 
                   B
2014-02-03 -1.658175

3、通过指定一个位置的列表，与numpy/python中的情况类似

In [34]: df.iloc[[1,2,4],[0,2]]
Out[34]: 
                   A         C
2013-01-02  1.212112  0.119209
2013-01-03 -0.861849 -0.494929
2013-01-05 -0.424972  0.276232

通过布尔索引

1、使用一个单独列的值来选择数据：

In [39]: df[df.A > 0]
Out[39]: 
                   A         B         C         D
2013-01-01  0.469112 -0.282863 -1.509059 -1.135632
2013-01-02  1.212112 -0.173215  0.119209 -1.044236
2013-01-04  0.721555 -0.706771 -1.039575  0.271860

2、使用isin()方法来过滤：

In[39]: data2
Out[39]: 
                   A         B         C         D
2014-02-01 -1.511038 -0.034441 -0.619328  0.921036
2014-02-02  0.936087  0.086909 -0.261515  0.627252
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594
2014-02-04 -1.609215  0.305436  0.695883 -1.213981
2014-02-05  0.159289 -0.748557  0.731476  1.809500
2014-02-06  0.966737  0.144001 -0.627874  0.251945
In[40]: data2['E']=['a']*2+['b']*2+['c']*2
In[41]: data2
Out[41]: 
                   A         B         C         D  E
2014-02-01 -1.511038 -0.034441 -0.619328  0.921036  a
2014-02-02  0.936087  0.086909 -0.261515  0.627252  a
2014-02-03 -2.010322 -1.658175 -1.214687  0.419594  b
2014-02-04 -1.609215  0.305436  0.695883 -1.213981  b
2014-02-05  0.159289 -0.748557  0.731476  1.809500  c
2014-02-06  0.966737  0.144001 -0.627874  0.251945  c
In[42]: data2[data2.E.isin(['a','c'])]
Out[42]: 
                   A         B         C         D  E
2014-02-01 -1.511038 -0.034441 -0.619328  0.921036  a
2014-02-02  0.936087  0.086909 -0.261515  0.627252  a
2014-02-05  0.159289 -0.748557  0.731476  1.809500  c
2014-02-06  0.966737  0.144001 -0.627874  0.251945  c

设置

1、设置一个新的列：

In [45]: s1 = pd.Series([1,2,3,4,5,6], index=pd.date_range('20130102', periods=6))
 
In [46]: s1
Out[46]: 
2013-01-02    1
2013-01-03    2
2013-01-04    3
2013-01-05    4
2013-01-06    5
2013-01-07    6
Freq: D, dtype: int64
 
In [47]: df['F'] = s1

2、通过标签设置新的值：

 In [48]: df.at[dates[0],'A'] = 0

3、通过位置设置新的值：

In [49]: df.iat[0,1] = 0

4、通过一个numpy数组设置一组新值：

In [50]: df.loc[:,'D'] = np.array([5] * len(df))

上述操作结果如下：

In [51]: df
Out[51]: 
                   A         B         C  D    F
2013-01-01  0.000000  0.000000 -1.509059  5  NaN
2013-01-02  1.212112 -0.173215  0.119209  5  1.0
2013-01-03 -0.861849 -2.104569 -0.494929  5  2.0
2013-01-04  0.721555 -0.706771 -1.039575  5  3.0
2013-01-05 -0.424972  0.567020  0.276232  5  4.0
2013-01-06 -0.673690  0.113648 -1.478427  5  5.0

5、通过where操作来设置新的值：

In [52]: df2 = df.copy()
 
In [53]: df2[df2 > 0] = -df2
 
In [54]: df2
Out[54]: 
                   A         B         C  D    F
2013-01-01  0.000000  0.000000 -1.509059 -5  NaN
2013-01-02 -1.212112 -0.173215 -0.119209 -5 -1.0
2013-01-03 -0.861849 -2.104569 -0.494929 -5 -2.0
2013-01-04 -0.721555 -0.706771 -1.039575 -5 -3.0
2013-01-05 -0.424972 -0.567020 -0.276232 -5 -4.0
2013-01-06 -0.673690 -0.113648 -1.478427 -5 -5.

四、缺失值处理

创建：

In[2]: import pandas as pd
In[3]: import numpy as np
In[4]: import matplotlib.pyplot as plt
In[5]: dates=pd.date_range('20160301',periods=6)
In[6]: df=pd.DataFrame(np.random.randn(6,4),index=dates,columns=list('ABCD'))
In[7]: df1=df.reindex(index=dates[0:4],columns=list(df.columns)+['E'])
In[8]: df1
Out[8]: 
                   A         B         C         D   E
2016-03-01  0.899779 -0.347769 -0.664284 -0.646859 NaN
2016-03-02 -0.019930 -0.679697  0.207176 -0.186060 NaN
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958 NaN
2016-03-04 -0.101724  0.160194 -0.802420  1.033333 NaN

1、 reindex()方法可以对指定轴上的索引进行改变/增加/删除操作，这将返回原始数据的一个拷贝：

df1=df.reindex(index=dates[0:4],columns=list(df.columns)+['E'])
df1.loc[dates[1:3],'E']=2

A	B	C	D	E
2016-03-01	0.141177	1.391996	0.732199	0.336293	NaN
2016-03-02	-0.661433	-0.659423	1.146476	-0.602315	2.0
2016-03-03	0.804278	-0.429059	-1.477499	0.514814	2.0
2016-03-04	1.649037	-0.071460	1.911577	-0.173560	NaN

2、去掉包含缺失值的行：

In[9]: df1.loc[dates[1:3],'E']=2
In[10]: df1
Out[10]: 
                   A         B         C         D    E
2016-03-01  0.899779 -0.347769 -0.664284 -0.646859  NaN
2016-03-02 -0.019930 -0.679697  0.207176 -0.186060  2.0
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958  2.0
2016-03-04 -0.101724  0.160194 -0.802420  1.033333  NaN
In[11]: df1.dropna()
Out[11]: 
                   A         B         C         D    E
2016-03-02 -0.019930 -0.679697  0.207176 -0.186060  2.0
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958  2.0

3、对缺失值进行填充：

In[13]: df1.fillna(value=3)
Out[13]: 
                   A         B         C         D    E
2016-03-01  0.899779 -0.347769 -0.664284 -0.646859  3.0
2016-03-02 -0.019930 -0.679697  0.207176 -0.186060  2.0
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958  2.0
2016-03-04 -0.101724  0.160194 -0.802420  1.033333  3.0

4、对数据进行布尔填充：

In[15]: pd.isnull(df1)
Out[15]: 
                A      B      C      D      E
2016-03-01  False  False  False  False   True
2016-03-02  False  False  False  False  False
2016-03-03  False  False  False  False  False
2016-03-04  False  False  False  False   True

5、判断是否有空数据

pd.isnull(df1).any().any()
Out[16]: True

五、相关操作

统计

1、执行描述性统计：

 In [17]: df.mean()
Out[17]: 
A   -0.004474
B   -0.383981
C   -0.687758
D    5.000000
F    3.000000
dtype: float64

2、对于拥有不同维度，需要对齐的对象进行操作。Pandas会自动的沿着指定的维度进行广播：

In[18]: s=pd.Series([1,3,5,np.nan,6,8],index=dates).shift(2)
In[19]: s
Out[19]: 
2016-03-01    NaN
2016-03-02    NaN
2016-03-03    1.0
2016-03-04    3.0
2016-03-05    5.0
2016-03-06    NaN
Freq: D, dtype: float64
In[20]: df
Out[20]: 
                   A         B         C         D
2016-03-01  0.899779 -0.347769 -0.664284 -0.646859
2016-03-02 -0.019930 -0.679697  0.207176 -0.186060
2016-03-03 -0.007573 -0.521074 -0.409476 -0.286958
2016-03-04 -0.101724  0.160194 -0.802420  1.033333
2016-03-05 -0.955321 -0.539865 -1.144294  0.114223
2016-03-06  0.798588 -1.888797 -0.053185  1.463853
In[21]: df.sub(s,axis='index')
Out[21]: 
                   A         B         C         D
2016-03-01       NaN       NaN       NaN       NaN
2016-03-02       NaN       NaN       NaN       NaN
2016-03-03 -1.007573 -1.521074 -1.409476 -1.286958
2016-03-04 -3.101724 -2.839806 -3.802420 -1.966667
2016-03-05 -5.955321 -5.539865 -6.144294 -4.885777
2016-03-06       NaN       NaN       NaN       NaN

apply

1、对数据应用函数：

/***累加****/
In[22]: df.apply(np.cumsum)
Out[22]: 
                   A         B         C         D
2016-03-01  0.899779 -0.347769 -0.664284 -0.646859
2016-03-02  0.879850 -1.027466 -0.457108 -0.832919
2016-03-03  0.872277 -1.548540 -0.866584 -1.119876
2016-03-04  0.770553 -1.388346 -1.669004 -0.086543
2016-03-05 -0.184767 -1.928211 -2.813298  0.027679
2016-03-06  0.613820 -3.817008 -2.866482  1.491532

/***最大值减最小值****/
In[23]: df.apply(lambda x:x.max()-x.min())
Out[23]: 
A    1.855100
B    2.048991
C    1.351470
D    2.110712
dtype: float64

次数显示

1、记录次数

In[5]: s=pd.Series(np.random.randint(0,7,size=10))
In[6]: s
Out[6]: 
0    6
1    4
2    0
3    2
4    4
5    0
6    6
7    4
8    1
9    4
dtype: int32
In[7]: s.value_counts()
Out[7]: 
4    4
6    2
0    2
2    1
1    1
dtype: int64

2、显示最多次数

s.mode()
Out[9]: 
0    4
dtype: int32

六、合并

Pandas提供了大量的方法能够轻松的对Series，DataFrame和Panel对象进行各种符合各种逻辑关系的合并操作。

pd .concat

In[12]: df
Out[12]: 
          A         B         C         D
0 -0.023904  0.370157 -0.189051 -1.228684
1 -0.209592 -0.348743 -1.136188 -1.103672
2 -1.014317 -1.806632  1.917631  0.450594
3 -1.035946 -0.359108 -0.570582  0.153201
4  1.667096  1.474152  1.624510 -1.513115
5 -1.091161  0.957389  0.447147 -2.581356
6  0.963045  0.575139  1.438793 -1.556901
7 -0.437601  0.451941 -0.137079 -0.050907
8 -0.629690 -0.556138  0.469743 -1.837013
9 -0.368121 -1.351038 -0.355778 -0.794521
In[13]: df.iloc[:3]
Out[13]: 
          A         B         C         D
0 -0.023904  0.370157 -0.189051 -1.228684
1 -0.209592 -0.348743 -1.136188 -1.103672
2 -1.014317 -1.806632  1.917631  0.450594
In[14]: df.iloc[3:7]
Out[14]: 
          A         B         C         D
3 -1.035946 -0.359108 -0.570582  0.153201
4  1.667096  1.474152  1.624510 -1.513115
5 -1.091161  0.957389  0.447147 -2.581356
6  0.963045  0.575139  1.438793 -1.556901
In[15]: df.iloc[7:]
Out[15]: 
          A         B         C         D
7 -0.437601  0.451941 -0.137079 -0.050907
8 -0.629690 -0.556138  0.469743 -1.837013
9 -0.368121 -1.351038 -0.355778 -0.794521
In[16]: df1=pd.concat([df.iloc[:3],df.iloc[3:7],df.iloc[7:]])
In[17]: 
In[17]: df1
Out[17]: 
          A         B         C         D
0 -0.023904  0.370157 -0.189051 -1.228684
1 -0.209592 -0.348743 -1.136188 -1.103672
2 -1.014317 -1.806632  1.917631  0.450594
3 -1.035946 -0.359108 -0.570582  0.153201
4  1.667096  1.474152  1.624510 -1.513115
5 -1.091161  0.957389  0.447147 -2.581356
6  0.963045  0.575139  1.438793 -1.556901
7 -0.437601  0.451941 -0.137079 -0.050907
8 -0.629690 -0.556138  0.469743 -1.837013
9 -0.368121 -1.351038 -0.355778 -0.794521

pd .merge

In[18]: left=pd.DataFrame({'key':['foo','foo'],'lval':[1,2]})
In[19]: right=pd.DataFrame({'key':['foo','foo'],'rval':[4,5]})
In[20]: left
Out[20]: 
   key  lval
0  foo     1
1  foo     2
In[21]: right
Out[21]: 
   key  rval
0  foo     4
1  foo     5
In[22]: pd.merge(left,right,on='key')
Out[22]: 
   key  lval  rval
0  foo     1     4
1  foo     1     5
2  foo     2     4
3  foo     2     5

Append 将一行连接到一个DataFrame上

In [82]: df = pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'])
 
In [83]: df
Out[83]: 
          A         B         C         D
0  1.346061  1.511763  1.627081 -0.990582
1 -0.441652  1.211526  0.268520  0.024580
2 -1.577585  0.396823 -0.105381 -0.532532
3  1.453749  1.208843 -0.080952 -0.264610
4 -0.727965 -0.589346  0.339969 -0.693205
5 -0.339355  0.593616  0.884345  1.591431
6  0.141809  0.220390  0.435589  0.192451
7 -0.096701  0.803351  1.715071 -0.708758
 
In [84]: s = df.iloc[3]
 
In [85]: df.append(s, ignore_index=True)
Out[85]: 
          A         B         C         D
0  1.346061  1.511763  1.627081 -0.990582
1 -0.441652  1.211526  0.268520  0.024580
2 -1.577585  0.396823 -0.105381 -0.532532
3  1.453749  1.208843 -0.080952 -0.264610
4 -0.727965 -0.589346  0.339969 -0.693205
5 -0.339355  0.593616  0.884345  1.591431
6  0.141809  0.220390  0.435589  0.192451
7 -0.096701  0.803351  1.715071 -0.708758
8  1.453749  1.208843 -0.080952 -0.264610

七、分组

按照一定的规则分组，并进行操作：

单索引

In[24]: df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
   ...:                             'foo', 'bar', 'foo', 'foo'],
   ...:                      'B' : ['one', 'one', 'two', 'three',
   ...:                             'two', 'two', 'one', 'three'],
   ...:                 'C' : np.random.randn(8),
   ...:                    'D' : np.random.randn(8)})
In[25]: df
Out[25]: 
     A      B         C         D
0  foo    one -0.133907  1.086158
1  bar    one -0.248065 -0.543809
2  foo    two -0.185406  0.856915
3  bar  three  0.732933 -0.727144
4  foo    two -1.222730  0.649772
5  bar    two -0.705964  0.852976
6  foo    one -1.357128 -0.236820
7  foo  three -0.455601 -0.163037
In[26]: df.groupby('A').sum()
Out[26]: 
            C         D
A                      
bar -0.221096 -0.417976
foo -3.354772  2.192987

双索引

In[28]: df.groupby(['A','B']).sum()
Out[28]: 
                  C         D
A   B                        
bar one   -0.248065 -0.543809
    three  0.732933 -0.727144
    two   -0.705964  0.852976
foo one   -1.491034  0.849337
    three -0.455601 -0.163037
    two   -1.408136  1.506687

八、重塑

** stack**

tuples=list(zip(*[['bar','bar','baz','baz','foo','foo','qux','qux'],
                  ['one','two','one','two','one','two','one','two']]))
index=pd.MultiIndex.from_tuples(tuples,names=['first','second'])
df=pd.DataFrame(np.random.randn(8,2),index=index,columns=['A','B'])
stacked=df.stack()

Out[12]:
first  second   
bar    one     A    0.204809
               B   -0.088132
       two     A   -0.499254
               B   -0.463513
baz    one     A   -0.149506
               B    1.153880
       two     A    2.227171
               B   -1.031684
foo    one     A    0.038368
               B   -0.763699
       two     A   -0.304946
               B    0.198953
qux    one     A   -0.535454
               B   -0.528770
       two     A   -1.305537
               B   -0.529129
dtype: float64

** stack()的逆运算**

stacked.unstack().unstack()
	A	B
second	one	two	one	two
first				
bar	0.204809	-0.499254	-0.088132	-0.463513
baz	-0.149506	2.227171	1.153880	-1.031684
foo	0.038368	-0.304946	-0.763699	0.198953
qux	-0.535454	-1.305537	-0.528770	-0.529129

数据透视

df = pd.DataFrame({'A' : ['one', 'one', 'two', 'three'] * 3,
                   'B' : ['A', 'B', 'C'] * 4,
                   'C' : ['foo', 'foo', 'foo', 'bar', 'bar', 'bar'] * 2,
                   'D' : np.random.randn(12),
                   'E' : np.random.randn(12)})
A	B	C	D	E
0	one	A	foo	0.260388	-1.040629
1	one	B	foo	1.204708	0.133854
2	two	C	foo	-1.000894	-0.522464
3	three	A	bar	-1.294604	0.434995
4	one	B	bar	0.226309	0.560666
5	one	C	bar	-0.572290	1.359220
6	two	A	foo	0.637389	-0.025343
7	three	B	foo	0.530362	-0.739961
8	one	C	foo	-0.463693	1.364281
9	one	A	bar	-0.134875	-1.148102
10	two	B	bar	-1.529663	0.062310
11	three	C	bar	1.414118	0.805989

df.pivot_table(values=['D'],index=['A','B'],columns=['C'])
		D
C	bar	foo
A	B		
one	A	-0.134875	0.260388
B	0.226309	1.204708
C	-0.572290	-0.463693
three	A	-1.294604	NaN
B	NaN	0.530362
C	1.414118	NaN
two	A	NaN	0.637389
B	-1.529663	NaN
C	NaN	-1.000894

当数据透视表对应多个值时求取得是平均值，对应空值时NAN

九、时间序列

Pandas在对频率转换进行重新采样时拥有简单、强大且高效的功能（如将按秒采样的数据转换为按2分钟为单位进行采样的数据）。这种操作在金融领域非常常见。

时间序列的创建

/****以2min为一个单位*****/
rng=pd.date_range('20160301',periods=600,freq='s')
DatetimeIndex(['2016-03-01 00:00:00', '2016-03-01 00:00:01',
               '2016-03-01 00:00:02', '2016-03-01 00:00:03',
               '2016-03-01 00:00:04', '2016-03-01 00:00:05',
               '2016-03-01 00:00:06', '2016-03-01 00:00:07',
               '2016-03-01 00:00:08', '2016-03-01 00:00:09',
               ...
               '2016-03-01 00:09:50', '2016-03-01 00:09:51',
               '2016-03-01 00:09:52', '2016-03-01 00:09:53',
               '2016-03-01 00:09:54', '2016-03-01 00:09:55',
               '2016-03-01 00:09:56', '2016-03-01 00:09:57',
               '2016-03-01 00:09:58', '2016-03-01 00:09:59'],
              dtype='datetime64[ns]', length=600, freq='S')
s=pd.Series(np.random.randint(0,500,len(rng)),index=rng)
s.resample('2Min',how='sum')
2016-03-01 00:00:00    30109
2016-03-01 00:02:00    32759
2016-03-01 00:04:00    32095
2016-03-01 00:06:00    30774
2016-03-01 00:08:00    29775
Freq: 2T, dtype: int32

In[29]: rng=pd.period_range('2000Q1','2016Q1',freq='Q')
In[30]: rng
Out[30]: 
PeriodIndex(['2000Q1', '2000Q2', '2000Q3', '2000Q4', '2001Q1', '2001Q2',
             '2001Q3', '2001Q4', '2002Q1', '2002Q2', '2002Q3', '2002Q4',
             '2003Q1', '2003Q2', '2003Q3', '2003Q4', '2004Q1', '2004Q2',
             '2004Q3', '2004Q4', '2005Q1', '2005Q2', '2005Q3', '2005Q4',
             '2006Q1', '2006Q2', '2006Q3', '2006Q4', '2007Q1', '2007Q2',
             '2007Q3', '2007Q4', '2008Q1', '2008Q2', '2008Q3', '2008Q4',
             '2009Q1', '2009Q2', '2009Q3', '2009Q4', '2010Q1', '2010Q2',
             '2010Q3', '2010Q4', '2011Q1', '2011Q2', '2011Q3', '2011Q4',
             '2012Q1', '2012Q2', '2012Q3', '2012Q4', '2013Q1', '2013Q2',
             '2013Q3', '2013Q4', '2014Q1', '2014Q2', '2014Q3', '2014Q4',
             '2015Q1', '2015Q2', '2015Q3', '2015Q4', '2016Q1'],
            dtype='period[Q-DEC]', freq='Q-DEC')

转化为时间日期格式：

In[31]: rng.to_timestamp()
Out[31]: 
DatetimeIndex(['2000-01-01', '2000-04-01', '2000-07-01', '2000-10-01',
               '2001-01-01', '2001-04-01', '2001-07-01', '2001-10-01',
               '2002-01-01', '2002-04-01', '2002-07-01', '2002-10-01',
               '2003-01-01', '2003-04-01', '2003-07-01', '2003-10-01',
               '2004-01-01', '2004-04-01', '2004-07-01', '2004-10-01',
               '2005-01-01', '2005-04-01', '2005-07-01', '2005-10-01',
               '2006-01-01', '2006-04-01', '2006-07-01', '2006-10-01',
               '2007-01-01', '2007-04-01', '2007-07-01', '2007-10-01',
               '2008-01-01', '2008-04-01', '2008-07-01', '2008-10-01',
               '2009-01-01', '2009-04-01', '2009-07-01', '2009-10-01',
               '2010-01-01', '2010-04-01', '2010-07-01', '2010-10-01',
               '2011-01-01', '2011-04-01', '2011-07-01', '2011-10-01',
               '2012-01-01', '2012-04-01', '2012-07-01', '2012-10-01',
               '2013-01-01', '2013-04-01', '2013-07-01', '2013-10-01',
               '2014-01-01', '2014-04-01', '2014-07-01', '2014-10-01',
               '2015-01-01', '2015-04-01', '2015-07-01', '2015-10-01',
               '2016-01-01'],
              dtype='datetime64[ns]', freq='QS-OCT')

日期相减

In[32]: pd.Timestamp('20160301')-pd.Timestamp('20160201')
Out[32]: Timedelta('29 days 00:00:00')

日期相加

十、Categorical

1、创建Categorical数据结构

In[33]: df=pd.DataFrame({'id':[1,2,3,4,5],'row_grade':['a','b','a','d','d']})
In[34]: df['grade']=df['row_grade'].astype('category')
In[35]: df
Out[35]: 
  id row_grade grade
0   1         a     a
1   2         b     b
2   3         a     a
3   4         d     d
4   5         d     d

2、将Categorical类型数据重命名为更有意义的名称：

df['grade'].cat.categories=['very good','good','bad']
df
Out[37]: 
   id row_grade      grade
0   1         a  very good
1   2         b       good
2   3         a  very good
3   4         d        bad
4   5         d        bad

3、排序是按照Categorical的顺序进行的而不是按照字典顺序进行：

In[38]: df.sort_values(by='grade')
Out[38]: 
   id row_grade      grade
0   1         a  very good
2   3         a  very good
1   2         b       good
3   4         d        bad
4   5         d        bad

十一、画图

t=pd.Series(np.random.randn(1000),index=pd.date_range(('20000101'),periods=1000))
t=t.cumsum()
t.plot()
plt.show()

在这里插入图片描述
对于DataFrame来说，plot是一种将所有列及其标签进行绘制的简便方法：

In [40]: df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,
   .....:                   columns=['A', 'B', 'C', 'D'])
   .....: 
 
In [41]: df = df.cumsum()
 
In [42]: plt.figure(); df.plot(); plt.legend(loc='best')
Out[42]: <matplotlib.legend.Legend at 0x112854d90

十二、导入和保存数据

CSV
1、写入CSV文件并读出

In[49]: df=pd.DataFrame(np.random.randn(100,4))
In[50]: df=pd.DataFrame(np.random.randn(100,4),columns=list('ABCD'))
In[51]: df
Out[51]: 
           A         B         C         D
0   0.378867 -0.091620  1.090019  0.999421
1   2.251850  0.238909 -0.901346  0.015554
2   0.209695 -2.048660 -0.955254  0.344090
3  -0.130926  0.164686  1.340037  2.832207
4   0.166250 -0.101414 -0.232638  0.456991
5   0.304952 -0.403781 -0.130427 -0.956354
6  -0.429590  0.773878 -0.996998 -0.292732
7  -0.473208  0.674366  0.960866  0.682791
8  -1.840992 -0.526458 -0.420458  1.559731
9  -0.068624  0.315819  1.037631 -0.446265
10  0.261860  0.364727  0.343445 -0.109746
11  0.311507 -0.532801 -1.150832  0.302602
12 -0.841183  0.589915 -0.161598  0.378613
13  1.419394  0.097689  0.224020 -0.720346
14  0.468242 -0.635890 -0.413119 -0.160551
15 -1.375062 -0.560435  2.428874 -1.007550
16 -0.063632  1.631429  1.021518 -0.053831
17 -1.696037 -0.328426  0.440562 -0.590583
18 -0.661396 -0.909790  0.087819  0.889130
19  2.111948 -1.750826  1.747665 -1.459147
20  1.205531 -0.862482  0.062849 -0.019009
21  1.743419 -0.564083  1.311440 -0.529935
22  0.016245 -0.102664 -1.099839 -1.010344
23 -0.482878  0.826072 -0.163725  0.097180
24 -0.139670  0.902446 -0.460139 -0.549083
25 -1.438988 -0.205327 -0.344405 -1.374123
26  0.308278 -1.064430  0.329222 -0.577498
27  0.733006  0.026945 -0.092934 -0.341411
28 -1.464381 -1.634260  0.909248 -1.700494
29  0.039253 -1.598910  1.284848  0.402043
..       ...       ...       ...       ...
70 -1.385963  1.633525 -0.422429 -0.886571
71 -1.471958 -0.001905  1.438166  0.150414
72  0.381426  0.403522  0.464480  0.191731
73 -0.280404  1.045549 -1.364438 -1.526277
74  0.458306 -0.996155 -1.699094 -1.194917
75 -0.307120 -2.056273 -1.043319  0.862359
76  1.864204  0.437932 -0.404072 -0.128899
77  1.271321 -0.747139  1.400846 -0.701221
78  1.589635  1.048788  0.706948  1.156733
79  1.378117 -0.179104 -2.180511  0.049153
80  0.555157  0.108048 -0.191770 -1.194258
81 -1.539549 -0.623444  2.785096 -0.408042
82 -1.062381 -0.007418  0.794200  0.129454
83 -0.177113  1.465207  0.538259  0.425892
84  0.053148 -1.966307 -0.165101  1.628191
85 -1.816071 -0.135639  0.147840 -0.383169
86 -0.328421 -0.063588 -0.039663  0.846303
87 -0.699656 -0.687535  0.586631  0.013061
88  0.074753 -0.858002 -0.647638  0.481088
89 -1.087874  0.883394 -0.654534 -1.213953
90 -1.460692 -0.038029 -0.706556  0.261179
91  0.759244 -1.255425  0.218774  0.058548
92  1.136994 -1.315591 -1.520542 -0.336128
93 -0.825148 -0.736311 -0.653980  0.506639
94  0.743375  0.457888  0.811990  1.193933
95 -0.055319 -0.188443 -1.926626 -0.186323
96 -0.277529  1.293196  0.026766 -1.764860
97  2.038703  0.785085  0.330018  0.056930
98 -0.532230 -1.306918 -0.666728  1.322078
99 -1.026237  1.315298  0.208255 -1.145776

[100 rows x 4 columns]
In[52]: df.to_csv('data.csv')
In[53]: pd.read_csv('data.csv',index_col=0)
Out[53]: 
           A         B         C         D
0   0.378867 -0.091620  1.090019  0.999421
1   2.251850  0.238909 -0.901346  0.015554
2   0.209695 -2.048660 -0.955254  0.344090
3  -0.130926  0.164686  1.340037  2.832207
4   0.166250 -0.101414 -0.232638  0.456991
5   0.304952 -0.403781 -0.130427 -0.956354
6  -0.429590  0.773878 -0.996998 -0.292732
7  -0.473208  0.674366  0.960866  0.682791
8  -1.840992 -0.526458 -0.420458  1.559731
9  -0.068624  0.315819  1.037631 -0.446265
10  0.261860  0.364727  0.343445 -0.109746
11  0.311507 -0.532801 -1.150832  0.302602
12 -0.841183  0.589915 -0.161598  0.378613
13  1.419394  0.097689  0.224020 -0.720346
14  0.468242 -0.635890 -0.413119 -0.160551
15 -1.375062 -0.560435  2.428874 -1.007550
16 -0.063632  1.631429  1.021518 -0.053831
17 -1.696037 -0.328426  0.440562 -0.590583
18 -0.661396 -0.909790  0.087819  0.889130
19  2.111948 -1.750826  1.747665 -1.459147
20  1.205531 -0.862482  0.062849 -0.019009
21  1.743419 -0.564083  1.311440 -0.529935
22  0.016245 -0.102664 -1.099839 -1.010344
23 -0.482878  0.826072 -0.163725  0.097180
24 -0.139670  0.902446 -0.460139 -0.549083
25 -1.438988 -0.205327 -0.344405 -1.374123
26  0.308278 -1.064430  0.329222 -0.577498
27  0.733006  0.026945 -0.092934 -0.341411
28 -1.464381 -1.634260  0.909248 -1.700494
29  0.039253 -1.598910  1.284848  0.402043
..       ...       ...       ...       ...
70 -1.385963  1.633525 -0.422429 -0.886571
71 -1.471958 -0.001905  1.438166  0.150414
72  0.381426  0.403522  0.464480  0.191731
73 -0.280404  1.045549 -1.364438 -1.526277
74  0.458306 -0.996155 -1.699094 -1.194917
75 -0.307120 -2.056273 -1.043319  0.862359
76  1.864204  0.437932 -0.404072 -0.128899
77  1.271321 -0.747139  1.400846 -0.701221
78  1.589635  1.048788  0.706948  1.156733
79  1.378117 -0.179104 -2.180511  0.049153
80  0.555157  0.108048 -0.191770 -1.194258
81 -1.539549 -0.623444  2.785096 -0.408042
82 -1.062381 -0.007418  0.794200  0.129454
83 -0.177113  1.465207  0.538259  0.425892
84  0.053148 -1.966307 -0.165101  1.628191
85 -1.816071 -0.135639  0.147840 -0.383169
86 -0.328421 -0.063588 -0.039663  0.846303
87 -0.699656 -0.687535  0.586631  0.013061
88  0.074753 -0.858002 -0.647638  0.481088
89 -1.087874  0.883394 -0.654534 -1.213953
90 -1.460692 -0.038029 -0.706556  0.261179
91  0.759244 -1.255425  0.218774  0.058548
92  1.136994 -1.315591 -1.520542 -0.336128
93 -0.825148 -0.736311 -0.653980  0.506639
94  0.743375  0.457888  0.811990  1.193933
95 -0.055319 -0.188443 -1.926626 -0.186323
96 -0.277529  1.293196  0.026766 -1.764860
97  2.038703  0.785085  0.330018  0.056930
98 -0.532230 -1.306918 -0.666728  1.322078
99 -1.026237  1.315298  0.208255 -1.145776

[100 rows x 4 columns]