Pandas入门
Numpy 和 Pandas有什么不同Numpy是列表形式的,没有index标签,而Pandas则类似于字典,会带标签。Pandas是基于Numpy构建的。
Pandas主要有俩个数据结构:Series和DataFrame
Series
<<< import pandas as pd
<<< import numpy as np
<<< s = pd.Series([1,3,6,np,nan,44,1])
<<< print(s)
"""
0 1.0
1 3.0
2 6.0
3 NaN
4 44.0
5 1.0
dtype: float64
"""
Series会出出现俩部分,左边的index部分和右边的value部分,在没有指定索引的情况下,会自动创建一个0到N-1的整数型索引。
DataFrame
<<< dates = pd.date_range('20160101', periods = 6)
<<< df = pd.DataFrame(np.random.randn(6,4), index = dates, columns = ['a','b','c','d'])
<<< print(df)
"""
a b c d
2016-01-01 -0.253065 -2.071051 -0.640515 0.613663
2016-01-02 -1.147178 1.532470 0.989255 -0.499761
2016-01-03 1.221656 -2.390171 1.862914 0.778070
2016-01-04 1.473877 -0.046419 0.610046 0.204672
2016-01-05 -1.584752 -0.700592 1.487264 -1.778293
2016-01-06 0.633675 -1.414157 -0.277066 -0.442545
"""
DataFrame 是表格形状的数据结构,包含一组有序的列,每一列可以由不同的值类型构成。并且既有行索引也有列索引,可以看作是由Series组成的大字典。我们可以根据每一个不同的索引来挑选数据。
<<< print(df['b'])
"""
2016-01-01 -2.071051
2016-01-02 1.532470
2016-01-03 -2.390171
2016-01-04 -0.046419
2016-01-05 -0.700592
2016-01-06 -1.414157
Freq: D, Name: b, dtype: float64
"""
当然也可以创建一组不指定行标签和列标签的数据df1:
<<< df1 = pd.DataFrame(np.arange(12).reshape((3,4)))
<<< print (df1)
"""
0 1 2 3
0 0 1 2 3
1 4 5 6 7
2 8 9 10 11
"""
这样,就会采用默认的0到N-1的方式来index,还有另外一种生成df的方法,例如df2:
<<< df2 = pd.DataFrame({'A':1.,
'B':pd.Timestamp('20130102),
'C':pd.Series(1,index = list(range(4)), dtype = 'float32'),
'D':np.array([3] * 4, dtype = 'int32'),
'E':pd..Categorical(["test", "train", "test", "train"]),
'F':'foo'})
<<< print(df2)
"""
A B C D E F
0 1.0 2013-01-02 1.0 3 test foo
1 1.0 2013-01-02 1.0 3 train foo
2 1.0 2013-01-02 1.0 3 test foo
3 1.0 2013-01-02 1.0 3 train foo
"""
如果想要查看数据类型,可以用dtype这个属性:
<<< print(df2.dtypes)
"""
df2.dtypes
A float64
B datetime64[ns]
C float32
D int32
E category
F object
dtype: object
"""
查看序号以及每种数据的名称以及值分别为:
<<< print(df2.index)
# Int64Index([0, 1, 2, 3], dtype='int64')
<<< print(df2.columns)
# Index(['A', 'B', 'C', 'D', 'E', 'F'], dtype='object')
<<< print(df2.values)
"""
array([[1.0, Timestamp('2013-01-02 00:00:00'), 1.0, 3, 'test', 'foo'],
[1.0, Timestamp('2013-01-02 00:00:00'), 1.0, 3, 'train', 'foo'],
[1.0, Timestamp('2013-01-02 00:00:00'), 1.0, 3, 'test', 'foo'],
[1.0, Timestamp('2013-01-02 00:00:00'), 1.0, 3, 'train', 'foo']], dtype=object)
"""
想查看数据的总结,则采用方法describe():
<<< df2.describe()
"""
A C D
count 4.0 4.0 4.0
mean 1.0 1.0 3.0
std 0.0 0.0 0.0
min 1.0 1.0 3.0
25% 1.0 1.0 3.0
50% 1.0 1.0 3.0
75% 1.0 1.0 3.0
max 1.0 1.0 3.0
"""
翻转数据也是采用:
<<< print(df2.T)
"""
0 1 2 \
A 1 1 1
B 2013-01-02 00:00:00 2013-01-02 00:00:00 2013-01-02 00:00:00
C 1 1 1
D 3 3 3
E test train test
F foo foo foo
3
A 1
B 2013-01-02 00:00:00
C 1
D 3
E train
F foo
"""
对index排序输出则用sort:
<<< print(df2.sort_index(axis = 1, ascending = False))
"""
F E D C B A
0 foo test 3 1.0 2013-01-02 1.0
1 foo train 3 1.0 2013-01-02 1.0
2 foo test 3 1.0 2013-01-02 1.0
3 foo train 3 1.0 2013-01-02 1.0
"""
对数据值排序输出则是:
<<< print(df2.sort_values(by='B'))
"""
A B C D E F
0 1.0 2013-01-02 1.0 3 test foo
1 1.0 2013-01-02 1.0 3 train foo
2 1.0 2013-01-02 1.0 3 test foo
3 1.0 2013-01-02 1.0 3 train foo
"""
Pandas选择数据
简单的选取DataFrame中的数据:
<<< print(df['A'])
<<< print(df.A)
"""
2013-01-01 0
2013-01-02 4
2013-01-03 8
2013-01-04 12
2013-01-05 16
2013-01-06 20
Freq: D, Name: A, dtype: int64
"""
选择跨越行则是:
<<< print(df[0:3])
"""
A B C D
2013-01-01 0 1 2 3
2013-01-02 4 5 6 7
2013-01-03 8 9 10 11
"""
<<< print(df['20130102':'20130104'])
"""
A B C D
2013-01-02 4 5 6 7
2013-01-03 8 9 10 11
2013-01-04 12 13 14 15
"""
如果df[3:3]将会是一个空的对象。后者选择20130102到20130104之间的数据,并且包括这俩个标签
我们还能通过loc来定义某行或者所有行,然后选其中某列或者所有列(":"代表所有)
<<< print(df.loc['20130102'])
"""
A 4
B 5
C 6
D 7
Name: 2013-01-02 00:00:00, dtype: int64
"""
<<< print(df.loc[:,['A','B']])
"""
A B
2013-01-01 0 1
2013-01-02 4 5
2013-01-03 8 9
2013-01-04 12 13
2013-01-05 16 17
2013-01-06 20 21
"""
<<< print(df.loc['20130102',['A','B']])
"""
A 4
B 5
Name: 2013-01-02 00:00:00, dtype: int64
"""
根据序列iloc来定位,可以选某一个,连续选,或者跨行选:
<<< print(df.iloc[3,1])
# 13
<<< print(df.iloc[3:5,1:3])
"""
B C
2013-01-04 13 14
2013-01-05 17 18
"""
<<< print(df.iloc[[1,3,5],1:3])
"""
B C
2013-01-02 5 6
2013-01-04 13 14
2013-01-06 21 22
"""
混合俩种选择方式的ix:
<<< print(df.ix[:3,['A','C']])
"""
A C
2013-01-01 0 2
2013-01-02 4 6
2013-01-03 8 10
"""
还可以通过判断指令选择,约束某项条件然后选择出当前所有数据:
<<< print(df[df.A>8])
"""
A B C D
2013-01-04 12 13 14 15
2013-01-05 16 17 18 19
2013-01-06 20 21 22 23
"""
Pandas设置值
可以根据位置设置loc和iloc来直接修改:
<<< df.iloc[2,2] = 1111
<<< df.loc['20130101','B'] = 2222
"""
A B C D
2013-01-01 0 2222 2 3
2013-01-02 4 5 6 7
2013-01-03 8 9 1111 11
2013-01-04 12 13 14 15
2013-01-05 16 17 18 19
2013-01-06 20 21 22 23
"""
或者根据条件来设置:
<<< df.B[df.A>4] = 0
"""
A B C D
2013-01-01 0 2222 2 3
2013-01-02 4 5 6 7
2013-01-03 8 0 1111 11
2013-01-04 12 0 14 15
2013-01-05 16 0 18 19
2013-01-06 20 0 22 23
"""
或者整行或者整列来设置:
<<< df['F'] = np.nan
"""
A B C D F
2013-01-01 0 2222 2 3 NaN
2013-01-02 4 5 6 7 NaN
2013-01-03 8 0 1111 11 NaN
2013-01-04 12 0 14 15 NaN
2013-01-05 16 0 18 19 NaN
2013-01-06 20 0 22 23 NaN
"""
也可以添加一个Series序列,但是长度必须要对齐:
<<< df['E'] = pd.Series([1,2,3,4,5,6], index=pd.date_range('20130101',periods=6))
"""
A B C D F E
2013-01-01 0 2222 2 3 NaN 1
2013-01-02 4 5 6 7 NaN 2
2013-01-03 8 0 1111 11 NaN 3
2013-01-04 12 0 14 15 NaN 4
2013-01-05 16 0 18 19 NaN 5
2013-01-06 20 0 22 23 NaN 6
"""
Pandas处理丢失的数据(NaN)
直接去掉含有NaN的行或者列,采用dropna()
<<< df.dropna(
<<< axis=0, # 0: 对行进行操作; 1: 对列进行操作
<<< how='any' # 'any': 只要存在 NaN 就 drop 掉; 'all': 必须全部是 NaN 才 drop
<<< )
"""
A B C D
2013-01-03 8 9.0 10.0 11
2013-01-04 12 13.0 14.0 15
2013-01-05 16 17.0 18.0 19
2013-01-06 20 21.0 22.0 23
"""
或者填充为其他值:
<<< df.fillna(value=0)
"""
A B C D
2013-01-01 0 0.0 2.0 3
2013-01-02 4 5.0 0.0 7
2013-01-03 8 9.0 10.0 11
2013-01-04 12 13.0 14.0 15
2013-01-05 16 17.0 18.0 19
2013-01-06 20 21.0 22.0 23
"""
也可以判断是否含有缺失数据NaN,为True表示含有
<<< df.isnull()
"""
A B C D
2013-01-01 False True False False
2013-01-02 False False True False
2013-01-03 False False False False
2013-01-04 False False False False
2013-01-05 False False False False
2013-01-06 False False False False
"""
<<< np.any(df.isnull()) == True
# True
Pandas导入与导出数据
csv、excel、json、html与pickle都是同样前缀代码.
import pandas as pd #加载模块
#读取csv
<<< data = pd.read_csv('student.csv')
#打印出data
<<< print(data)
<<< data.to_pickle('student.pickle')
Pandas合并
axis=0是默认值,因此未设定任何参数时,函数默认axis=0。
<<< import pandas as pd
<<< import numpy as np
#定义资料集
<<< df1 = pd.DataFrame(np.ones((3,4))*0, columns=['a','b','c','d'])
<<< df2 = pd.DataFrame(np.ones((3,4))*1, columns=['a','b','c','d'])
<<< df3 = pd.DataFrame(np.ones((3,4))*2, columns=['a','b','c','d'])
#concat纵向合并
<<< res = pd.concat([df1, df2, df3], axis=0)
#打印结果
<<< print(res)
# a b c d
# 0 0.0 0.0 0.0 0.0
# 1 0.0 0.0 0.0 0.0
# 2 0.0 0.0 0.0 0.0
# 0 1.0 1.0 1.0 1.0
# 1 1.0 1.0 1.0 1.0
# 2 1.0 1.0 1.0 1.0
# 0 2.0 2.0 2.0 2.0
# 1 2.0 2.0 2.0 2.0
# 2 2.0 2.0 2.0 2.0
但是我们发现,他的index值会重叠,所以如果需要重置,则使用:
#承上一个例子,并将index_ignore设定为True
<<< res = pd.concat([df1, df2, df3], axis=0, ignore_index=True)
#打印结果
<<< print(res)
# a b c d
# 0 0.0 0.0 0.0 0.0
# 1 0.0 0.0 0.0 0.0
# 2 0.0 0.0 0.0 0.0
# 3 1.0 1.0 1.0 1.0
# 4 1.0 1.0 1.0 1.0
# 5 1.0 1.0 1.0 1.0
# 6 2.0 2.0 2.0 2.0
# 7 2.0 2.0 2.0 2.0
# 8 2.0 2.0 2.0 2.0
而怎么根据关键字key来合并和,这个用于数据表中比较多:
<<< import pandas as pd
#定义资料集并打印出
<<< left = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3']})
<<< right = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
'C': ['C0', 'C1', 'C2', 'C3'],
'D': ['D0', 'D1', 'D2', 'D3']})
<<< print(left)
# A B key
# 0 A0 B0 K0
# 1 A1 B1 K1
# 2 A2 B2 K2
# 3 A3 B3 K3
<<< print(right)
# C D key
# 0 C0 D0 K0
# 1 C1 D1 K1
# 2 C2 D2 K2
# 3 C3 D3 K3
#依据key column合并,并打印出
<<< res = pd.merge(left, right, on='key')
<<< print(res)
A B key C D
# 0 A0 B0 K0 C0 D0
# 1 A1 B1 K1 C1 D1
# 2 A2 B2 K2 C2 D2
# 3 A3 B3 K3 C3 D3
而合并其实有4中how = ['left', 'right', 'outer', 'inner'],预设值how='inner'。
依据俩组key来合并则需要:
#定义资料集并打印出
<<< left = pd.DataFrame({'key1': ['K0', 'K0', 'K1', 'K2'],
'key2': ['K0', 'K1', 'K0', 'K1'],
'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3']})
<<< right = pd.DataFrame({'key1': ['K0', 'K1', 'K1', 'K2'],
'key2': ['K0', 'K0', 'K0', 'K0'],
'C': ['C0', 'C1', 'C2', 'C3'],
'D': ['D0', 'D1', 'D2', 'D3']})
<<< print(left)
# A B key1 key2
# 0 A0 B0 K0 K0
# 1 A1 B1 K0 K1
# 2 A2 B2 K1 K0
# 3 A3 B3 K2 K1
<<< print(right)
# C D key1 key2
# 0 C0 D0 K0 K0
# 1 C1 D1 K1 K0
# 2 C2 D2 K1 K0
# 3 C3 D3 K2 K0
#依据key1与key2 columns进行合并,并打印出四种结果['left', 'right', 'outer', 'inner']
<<< res = pd.merge(left, right, on=['key1', 'key2'], how='inner')
<<< print(res)
# A B key1 key2 C D
# 0 A0 B0 K0 K0 C0 D0
# 1 A2 B2 K1 K0 C1 D1
# 2 A2 B2 K1 K0 C2 D2
<<< res = pd.merge(left, right, on=['key1', 'key2'], how='outer')
<<< print(res)
# A B key1 key2 C D
# 0 A0 B0 K0 K0 C0 D0
# 1 A1 B1 K0 K1 NaN NaN
# 2 A2 B2 K1 K0 C1 D1
# 3 A2 B2 K1 K0 C2 D2
# 4 A3 B3 K2 K1 NaN NaN
# 5 NaN NaN K2 K0 C3 D3
<<< res = pd.merge(left, right, on=['key1', 'key2'], how='left')
<<< print(res)
# A B key1 key2 C D
# 0 A0 B0 K0 K0 C0 D0
# 1 A1 B1 K0 K1 NaN NaN
# 2 A2 B2 K1 K0 C1 D1
# 3 A2 B2 K1 K0 C2 D2
# 4 A3 B3 K2 K1 NaN NaN
<<< res = pd.merge(left, right, on=['key1', 'key2'], how='right')
<<< print(res)
# A B key1 key2 C D
# 0 A0 B0 K0 K0 C0 D0
# 1 A2 B2 K1 K0 C1 D1
# 2 A2 B2 K1 K0 C2 D2
# 3 NaN NaN K2 K0 C3 D3
而怎么根据index来合并呢?
#定义资料集并打印出
<<< left = pd.DataFrame({'A': ['A0', 'A1', 'A2'],
'B': ['B0', 'B1', 'B2']},
index=['K0', 'K1', 'K2'])
<<< right = pd.DataFrame({'C': ['C0', 'C2', 'C3'],
'D': ['D0', 'D2', 'D3']},
index=['K0', 'K2', 'K3'])
<<< print(left)
# A B
# K0 A0 B0
# K1 A1 B1
# K2 A2 B2
<<< print(right)
# C D
# K0 C0 D0
# K2 C2 D2
# K3 C3 D3
#依据左右资料集的index进行合并,how='outer',并打印出
<<< res = pd.merge(left, right, left_index=True, right_index=True, how='outer')
<<< print(res)
# A B C D
# K0 A0 B0 C0 D0
# K1 A1 B1 NaN NaN
# K2 A2 B2 C2 D2
# K3 NaN NaN C3 D3
#依据左右资料集的index进行合并,how='inner',并打印出
<<< res = pd.merge(left, right, left_index=True, right_index=True, how='inner')
<<< print(res)
# A B C D
# K0 A0 B0 C0 D0
# K2 A2 B2 C2 D2
解决overlapping问题:
<<< import pandas as pd
#定义资料集
<<< boys = pd.DataFrame({'k': ['K0', 'K1', 'K2'], 'age': [1, 2, 3]})
<<< girls = pd.DataFrame({'k': ['K0', 'K0', 'K3'], 'age': [4, 5, 6]})
使用suffixes解决overlapping的问题
<<< res = pd.merge(boys, girls, on='k', suffixes=['_boy', '_girl'], how='inner')
<<< print(res)
# age_boy k age_girl
# 0 1 K0 4
# 1 1 K0 5