目录
前言
莫烦老师Pandas教程，将所有代码和对应的输出记录在博客中，方便自己后续查看
作者：莫烦Python
转自：https://mofanpy.com/tutorials/data-manipulation/pandas/
视频：[【莫烦Python】Numpy & Pandas(数据处理教程)]
代码：https://github.com/MorvanZhou/tutorials/tree/master/numpy%26pandas
环境：python-3.9.13 pandas-1.4.4
1.Pandas vs Numpy
numpy是以矩阵为基础的数据计算模块，是数值计算的扩展包
pandas主要做数据处理，提供了DataFrame的数据结构，契合统计分析的表结构，可用numpy进行计算
numpy是python中的列表，pandas是python中的字典
pandas记录的信息可以特别丰富，另外pandas用于处理数据的功能也比较多，信息种类也更丰富，但是运行速度稍微比numpy慢
在做少量的数据分析时，因为不涉及到机器学习的模型运算等，可以用Pandas，但是如果要模型训练，训练过程中还一直调用数据处理的功能，则可选用Numpy来做
总结：Pandas是Numpy的封装库，继承了Numpy的很多优良传统，也具备了丰富的功能组件，但是得分情况来酌情选择要使用的工具。
2.基本介绍
Pandas基于两种数据类型：series和dataframe
- Series：一种类似于一维数组的对象，是由一组数据以及一组与之相关的数据标签(即索引)组成
- DataFrame：一个表格型的数据结构，包含一组有序的列，每列可以是不同的值类型(数值、字符串、布尔型等)，DataFrame既有行索引也有列索引，可以被看做是由Series组成的字典
import pandas as pd
import numpy as np
# Series数据类型
s = pd.Series([1, 3, 6, np.nan, 44, 1])
print("s:\n", s, "\n")
# DataFrame数据类型及创建方式
dates = pd.date_range('20230106', periods=6)
print("dates:\n", dates, "\n")
df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=['a', 'b', 'c', 'd'])
print("df:\n", df, "\n")
df1 = pd.DataFrame(np.arange(12).reshape(3,4))
print("df1:\n", df1, "\n")
df2 = pd.DataFrame({'A' : 1.,
'B' : pd.Timestamp('20130102'),
'C' : pd.Series(1,index=list(range(4)),dtype='float32'),
'D' : np.array([3] * 4,dtype='int32'),
'E' : pd.Categorical(["test","train","test","train"]),
'F' : 'foo'})
print("df2:\n", df2, "\n")
# DataFrame属性
print("df2.dtypes:\n", df2.dtypes, "\n") # 数据类型
print("df2.index:\n", df2.index, "\n") # 每一行
print("df2.columns:\n", df2.columns, "\n") # 每一列
print("df2.values:\n", df2.values, "\n") # NDFrame的Numpy表示
print("df2.describe:\n", df2.describe(), "\n") # 运算数字
print("df2.T:\n", df2.T, "\n")
print("df2.sort_index:\n", df2.sort_index(axis=1, ascending=False), "\n") # 排序
print("df2.sort_values:\n", df2.sort_values(by='E'), "\n")
运行结果如下：
s:
0 1.0
1 3.0
2 6.0
3 NaN
4 44.0
5 1.0
dtype: float64
dates:
DatetimeIndex(['2023-01-06', '2023-01-07', '2023-01-08', '2023-01-09',
'2023-01-10', '2023-01-11'],
dtype='datetime64[ns]', freq='D')
df:
a b c d
2023-01-06 -1.161849 -1.337222 -1.909200 -0.646599
2023-01-07 0.399974 1.303316 0.984053 0.276056
2023-01-08 1.902331 -2.203783 1.081992 0.107517
2023-01-09 1.009825 0.493630 -0.075483 -1.189449
2023-01-10 0.438570 0.399921 -0.329971 -0.592417
2023-01-11 1.082420 -0.428190 0.929953 -0.821798
df1:
0 1 2 3
0 0 1 2 3
1 4 5 6 7
2 8 9 10 11
df2:
A B C D E F
0 1.0 2013-01-02 1.0 3 test foo
1 1.0 2013-01-02 1.0 3 train foo
2 1.0 2013-01-02 1.0 3 test foo
3 1.0 2013-01-02 1.0 3 train foo
df2.dtypes:
A float64
B datetime64[ns]
C float32
D int32
E category
F object
dtype: object
df2.index:
Int64Index([0, 1, 2, 3], dtype='int64')
df2.columns:
Index(['A', 'B', 'C', 'D', 'E', 'F'], dtype='object')
df2.values:
[[1.0 Timestamp('2013-01-02 00:00:00') 1.0 3 'test' 'foo']
[1.0 Timestamp('2013-01-02 00:00:00') 1.0 3 'train' 'foo']
[1.0 Timestamp('2013-01-02 00:00:00') 1.0 3 'test' 'foo']
[1.0 Timestamp('2013-01-02 00:00:00') 1.0 3 'train' 'foo']]
df2.describe:
A C D
count 4.0 4.0 4.0
mean 1.0 1.0 3.0
std 0.0 0.0 0.0
min 1.0 1.0 3.0
25% 1.0 1.0 3.0
50% 1.0 1.0 3.0
75% 1.0 1.0 3.0
max 1.0 1.0 3.0
df2.T:
0 1 2 3
A 1.0 1.0 1.0 1.0
B 2013-01-02 00:00:00 2013-01-02 00:00:00 2013-01-02 00:00:00 2013-01-02 00:00:00
C 1.0 1.0 1.0 1.0
D 3 3 3 3
E test train test train
F foo foo foo foo
df2.sort_index:
F E D C B A
0 foo test 3 1.0 2013-01-02 1.0
1 foo train 3 1.0 2013-01-02 1.0
2 foo test 3 1.0 2013-01-02 1.0
3 foo train 3 1.0 2013-01-02 1.0
df2.sort_values:
A B C D E F
0 1.0 2013-01-02 1.0 3 test foo
2 1.0 2013-01-02 1.0 3 test foo
1 1.0 2013-01-02 1.0 3 train foo
3 1.0 2013-01-02 1.0 3 train foo
3.选择数据
在Pandas中，有丰富的选取数据方式，主要分为以下几种：
-
选Column or Row(index)
-
loc
-
iloc
-
条件过滤筛选
import pandas as pd
import numpy as np
dates = pd.date_range('20230106', periods=6)
df = pd.DataFrame(np.arange(24).reshape((6, 4)), index=dates, columns=['A', 'B', 'C', 'D'])
# select by label:Column or Row(index)
print("df:\n", df, "\n")
print("df['A'] or df.A:\n", df['A'], df.A, "\n")
print("df[0:3]:\n", df[0:3], "\n", "df['20230107':'20230108']:\n", df['20230107':'20230108'], "\n")
# select by label:loc
print("df.loc['20230106']:\n", df.loc['20230106'], "\n")
print("df.loc[:,['A', 'B']]:\n", df.loc[:,['A', 'B']], "\n")
print("df.loc['20230106',['A', 'B']]:\n", df.loc['20230106',['A', 'B']], "\n")
# select by position:iloc
print("df.iloc[3:5, 1:3]:\n", df.iloc[3:5, 1:3], "\n")
# boolean indexing
print("df[df.A > 8]:\n", df[df.A > 8], "\n")
运行结果如下：
df:
A B C D
2023-01-06 0 1 2 3
2023-01-07 4 5 6 7
2023-01-08 8 9 10 11
2023-01-09 12 13 14 15
2023-01-10 16 17 18 19
2023-01-11 20 21 22 23
df['A'] or df.A:
2023-01-06 0
2023-01-07 4
2023-01-08 8
2023-01-09 12
2023-01-10 16
2023-01-11 20
Freq: D, Name: A, dtype: int32 2023-01-06 0
2023-01-07 4
2023-01-08 8
2023-01-09 12
2023-01-10 16
2023-01-11 20
Freq: D, Name: A, dtype: int32
df[0:3]:
A B C D
2023-01-06 0 1 2 3
2023-01-07 4 5 6 7
2023-01-08 8 9 10 11
df['20230107':'20230108']:
A B C D
2023-01-07 4 5 6 7
2023-01-08 8 9 10 11
df.loc['20230106']:
A 0
B 1
C 2
D 3
Name: 2023-01-06 00:00:00, dtype: int32
df.loc[:,['A', 'B']]:
A B
2023-01-06 0 1
2023-01-07 4 5
2023-01-08 8 9
2023-01-09 12 13
2023-01-10 16 17
2023-01-11 20 21
df.loc['20230106',['A', 'B']]:
A 0
B 1
Name: 2023-01-06 00:00:00, dtype: int32
df.iloc[3:5, 1:3]:
B C
2023-01-09 13 14
2023-01-10 17 18
df[df.A > 8]:
A B C D
2023-01-09 12 13 14 15
2023-01-10 16 17 18 19
2023-01-11 20 21 22 23
4.设置值
Pandas设置值可以先通过前面的几种方式选取完数据后进行修改
import pandas as pd
import numpy as np
dates = pd.date_range('20230106', periods=6)
df = pd.DataFrame(np.arange(24).reshape((6, 4)), index=dates, columns=['A', 'B', 'C', 'D'])
print('修改前的df:\n', df, "\n")
df.iloc[2, 2] = 666
df.loc["20230106", "B"] = 222
df.A[df.A > 4] = 0
df['E'] = np.nan
df['F'] = pd.Series([1,2,3,4,5,6], index=dates)
print('修改后的df:\n', df, "\n")
运行结果如下：
修改前的df:
A B C D
2023-01-06 0 1 2 3
2023-01-07 4 5 6 7
2023-01-08 8 9 10 11
2023-01-09 12 13 14 15
2023-01-10 16 17 18 19
2023-01-11 20 21 22 23
修改后的df:
A B C D E F
2023-01-06 0 222 2 3 NaN 1
2023-01-07 4 5 6 7 NaN 2
2023-01-08 0 9 666 11 NaN 3
2023-01-09 0 13 14 15 NaN 4
2023-01-10 0 17 18 19 NaN 5
2023-01-11 0 21 22 23 NaN 6
5.处理丢失的数据
- 移除Nan的数据
- df.dropna
- 填充Nan的数据
- df.fillna
- 判断是否包含Nan的数据
- df.isnull()
import pandas as pd
import numpy as np
dates = pd.date_range('20230106', periods=6)
df = pd.DataFrame(np.arange(24).reshape((6, 4)), index=dates, columns=['A', 'B', 'C', 'D'])
df.iloc[0,1] = np.nan
df.iloc[1,2] = np.nan
print(df.dropna(axis=0, how='all')) # how = {'any', 'all}
print(df.fillna(value=0))
print(np.any(df.isnull()) == True)
运行结果如下：
A B C D
2023-01-06 0 NaN 2.0 3
2023-01-07 4 5.0 NaN 7
2023-01-08 8 9.0 10.0 11
2023-01-09 12 13.0 14.0 15
2023-01-10 16 17.0 18.0 19
2023-01-11 20 21.0 22.0 23
A B C D
2023-01-06 0 0.0 2.0 3
2023-01-07 4 5.0 0.0 7
2023-01-08 8 9.0 10.0 11
2023-01-09 12 13.0 14.0 15
2023-01-10 16 17.0 18.0 19
2023-01-11 20 21.0 22.0 23
True
6.pandas导入导出
pandas导入导出文件主要有以下几种：
- Excel文件
- pd.read_excel()
- df.to_excel()
- csv或txt等纯文本文件
- pd.read_csv()
- df.to_csv()
- 其他
- pd.read_clipboard()
- pd.read_html()
import pandas as pd
# read from
data = pd.read_csv("student.csv")
print(data)
# save to
data.to_pickle("student.pickle")
运行结果如下：
Student ID name age gender
0 1100 Kelly 22 Female
1 1101 Clo 21 Female
2 1102 Tilly 22 Female
3 1103 Tony 24 Male
4 1104 David 20 Male
5 1105 Catty 22 Female
6 1106 M 3 Female
7 1107 N 43 Male
8 1108 A 13 Male
9 1109 S 12 Male
10 1110 David 33 Male
11 1111 Dw 3 Female
12 1112 Q 23 Male
13 1113 W 21 Female
7.pandas合并concat
import pandas as pd
import numpy as np
# concatenating ignore_index
df1 = pd.DataFrame(np.ones((3,4))*0, columns=['a','b','c','d'])
df2 = pd.DataFrame(np.ones((3,4))*1, columns=['a','b','c','d'])
df3 = pd.DataFrame(np.ones((3,4))*2, columns=['a','b','c','d'])
print("df1:\n", df1, "\n")
print("df2:\n", df2, "\n")
print("df3:\n", df3, "\n")
res = pd.concat([df1, df2, df3], axis=0, ignore_index=True)
print("res:\n", res, "\n")
# join, ['inner', 'outer']
df1 = pd.DataFrame(np.ones((3,4))*0, columns=['a','b','c','d'], index=[1,2,3])
df2 = pd.DataFrame(np.ones((3,4))*1, columns=['b','c','d', 'e'], index=[2,3,4])
print("df1:\n", df1, "\n")
print("df2:\n", df2, "\n")
res1 = pd.concat([df1, df2], join='outer')
res2 = pd.concat([df1, df2], join='inner', ignore_index=True)
print("res1:\n", res1, "\n")
print("res2:\n", res2, "\n")
运行结果如下：
df1:
a b c d
0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0
df2:
a b c d
0 1.0 1.0 1.0 1.0
1 1.0 1.0 1.0 1.0
2 1.0 1.0 1.0 1.0
df3:
a b c d
0 2.0 2.0 2.0 2.0
1 2.0 2.0 2.0 2.0
2 2.0 2.0 2.0 2.0
res:
a b c d
0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0
3 1.0 1.0 1.0 1.0
4 1.0 1.0 1.0 1.0
5 1.0 1.0 1.0 1.0
6 2.0 2.0 2.0 2.0
7 2.0 2.0 2.0 2.0
8 2.0 2.0 2.0 2.0
df1:
a b c d
1 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0
df2:
b c d e
2 1.0 1.0 1.0 1.0
3 1.0 1.0 1.0 1.0
4 1.0 1.0 1.0 1.0
res1:
a b c d e
1 0.0 0.0 0.0 0.0 NaN
2 0.0 0.0 0.0 0.0 NaN
3 0.0 0.0 0.0 0.0 NaN
2 NaN 1.0 1.0 1.0 1.0
3 NaN 1.0 1.0 1.0 1.0
4 NaN 1.0 1.0 1.0 1.0
res2:
b c d
0 0.0 0.0 0.0
1 0.0 0.0 0.0
2 0.0 0.0 0.0
3 1.0 1.0 1.0
4 1.0 1.0 1.0
5 1.0 1.0 1.0
8.pandas合并merge
concat可以一次性合并多个df，可以左右，也可以上下拼接，但是merge是用来针对两张df做左右拼接的
import pandas as pd
# merging two df by key/keys. (may be used in database)
# simple example
print("==========simple example==========")
left = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3']})
right = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3'],
'C': ['C0', 'C1', 'C2', 'C3'],
'D': ['D0', 'D1', 'D2', 'D3']})
print("left:\n", left, "\n")
print("right:\n", right, "\n")
res = pd.merge(left, right, on='key')
print("res:\n", res, "\n")
# consider two keys
print("==========consider two keys==========")
left = pd.DataFrame({'key1': ['K0', 'K0', 'K1', 'K2'],
'key2': ['K0', 'K1', 'K0', 'K1'],
'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3']})
right = pd.DataFrame({'key1': ['K0', 'K1', 'K1', 'K2'],
'key2': ['K0', 'K0', 'K0', 'K0'],
'C': ['C0', 'C1', 'C2', 'C3'],
'D': ['D0', 'D1', 'D2', 'D3']})
print("left:\n", left, "\n")
print("right:\n", right, "\n")
# how = ['left', 'right', 'outer', 'inner']
res = pd.merge(left, right, on=['key1', 'key2'], how='inner') # 默认是inner
print("res:\n", res, "\n")
# indicator(显示怎样进行merge的)
print("==========indicator==========")
df1 = pd.DataFrame({'col1':[0,1], 'col_left':['a','b']})
df2 = pd.DataFrame({'col1':[1,2,2],'col_right':[2,2,2]})
print("df1:\n", df1, "\n")
print("df2:\n", df2, "\n")
res = pd.merge(df1, df2, on='col1', how='outer', indicator=True) # 默认是False
print("res:\n", res, "\n")
res = pd.merge(df1, df2, on='col1', how='outer', indicator='indicator_column')
# merged by index
print("==========merged by index==========")
left = pd.DataFrame({'A': ['A0', 'A1', 'A2'],
'B': ['B0', 'B1', 'B2']},
index=['K0', 'K1', 'K2'])
right = pd.DataFrame({'C': ['C0', 'C2', 'C3'],
'D': ['D0', 'D2', 'D3']},
index=['K0', 'K2', 'K3'])
print("left:\n", left, "\n")
print("right:\n", right, "\n")
# left_index right_index
res = pd.merge(left, right, left_index=True, right_index=True, how="outer")
print("res:\n", res, "\n")
res = pd.merge(left, right, left_index=True, right_index=True, how='inner')
# handle overlapping
print("==========handle overlapping==========")
boys = pd.DataFrame({'k': ['K0', 'K1', 'K2'], 'age': [1, 2, 3]})
girls = pd.DataFrame({'k': ['K0', 'K0', 'K3'], 'age': [4, 5, 6]})
print("boys:\n", boys, "\n")
print("girls:\n", girls, "\n")
res = pd.merge(boys, girls, on='k', suffixes=['_boy', '_girl'], how='inner')
print("res:\n", res, "\n")
运行结果如下：
==========simple example==========
left:
key A B
0 K0 A0 B0
1 K1 A1 B1
2 K2 A2 B2
3 K3 A3 B3
right:
key C D
0 K0 C0 D0
1 K1 C1 D1
2 K2 C2 D2
3 K3 C3 D3
res:
key A B C D
0 K0 A0 B0 C0 D0
1 K1 A1 B1 C1 D1
2 K2 A2 B2 C2 D2
3 K3 A3 B3 C3 D3
==========consider two keys==========
left:
key1 key2 A B
0 K0 K0 A0 B0
1 K0 K1 A1 B1
2 K1 K0 A2 B2
3 K2 K1 A3 B3
right:
key1 key2 C D
0 K0 K0 C0 D0
1 K1 K0 C1 D1
2 K1 K0 C2 D2
3 K2 K0 C3 D3
res:
key1 key2 A B C D
0 K0 K0 A0 B0 C0 D0
1 K1 K0 A2 B2 C1 D1
2 K1 K0 A2 B2 C2 D2
==========indicator==========
df1:
col1 col_left
0 0 a
1 1 b
df2:
col1 col_right
0 1 2
1 2 2
2 2 2
res:
col1 col_left col_right _merge
0 0 a NaN left_only
1 1 b 2.0 both
2 2 NaN 2.0 right_only
3 2 NaN 2.0 right_only
==========merged by index==========
left:
A B
K0 A0 B0
K1 A1 B1
K2 A2 B2
right:
C D
K0 C0 D0
K2 C2 D2
K3 C3 D3
res:
A B C D
K0 A0 B0 C0 D0
K1 A1 B1 NaN NaN
K2 A2 B2 C2 D2
K3 NaN NaN C3 D3
==========handle overlapping==========
boys:
k age
0 K0 1
1 K1 2
2 K2 3
girls:
k age
0 K0 4
1 K0 5
2 K3 6
res:
k age_boy age_girl
0 K0 1 4
1 K0 1 5
9.pandas plot画图
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# plot data
# Series
data = pd.Series(np.random.randn(1000), index=np.arange(1000))
data = data.cumsum()
data.plot()
plt.show()
# DataFrame
data = pd.DataFrame(np.random.randn(1000, 4),
index=np.arange(1000),
columns=list("ABCD"))
data = data.cumsum()
data.plot()
plt.show()
# plot methods:
# 'bar', 'hist', 'box', 'kde', 'area', scatter', 'hexbin', 'pie'
ax = data.plot.scatter(x='A', y='B', color='DarkBlue', label="Class 1")
data.plot.scatter(x='A', y='C', color='LightGreen', label='Class 2', ax=ax)
plt.show()
运行效果如下：
结语
代码仅供自己参考，大家可以查看对应的教程视频自行学习
694
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
