10 分钟 pandas
1 对象创建
pandas 中有两种对象, Series 和 DataFrame
# Series
s = pd.Series([1, 3, 5, np.nan, 6, 8])
# DataFrame
dates = pd.date_range("20130101", periods=6)
df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list("ABCD"))
Out[8]:
A B C D
2013-01-01 0.469112 -0.282863 -1.509059 -1.135632
2013-01-02 1.212112 -0.173215 0.119209 -1.044236
2013-01-03 -0.861849 -2.104569 -0.494929 1.071804
2013-01-04 0.721555 -0.706771 -1.039575 0.271860
2013-01-05 -0.424972 0.567020 0.276232 -1.087401
2013-01-06 -0.673690 0.113648 -1.478427 0.524988
2 数据可视化
df.head()
df.tail(3)
df.index
df.columns
df.describe()
DataFrame 类型转为 Numpy 类型,DataFrame.to_numpy() 。Numpy 数组全部数据为一种类型,DataFrame 中列数据为一种类型。所以当 DataFrame 列有多种数据类型时,会将数据转到一种泛化类型
# 转置
df.T
# 按轴排序
df.sort_index(axis=1, ascending=False)
# 按某列值排序
df.sort_values(by="B")
3 数据选择
3.1 获取数据
# 选择某一列,返回 Series
df["A"]
# slices
df[0:3]
3.2 根据标签获取数据
# 通过轴标签选择
df.loc[dates[0]]
df.loc[:, ["A", "B"]]
# 得到标量 等价于 df.at[dates[0], "A"] 速度更快
df.loc[dates[0], "A"]
3.3 根据位置获取数据
loc根据标签值获取数据,iloc则根据位置获取数据
df.iloc[3]
# slices
df.iloc[3:5, 0:2]
# 特定行列
df.iloc[[1, 2, 4], [0, 2]]
# 特定行,所有列
df.iloc[1:3, :]
# 特定值,标量
df.iloc[1, 1]
3.4 布尔索引
# 对某列值做限制
df[df["A"] > 0]
# isin() 方法
df2 = df.copy()
df2["E"] = ["one", "one", "two", "three", "four", "three"]
df2[df2["E"].isin(["two", "four"])]
Out[44]:
A B C D E
2013-01-03 -0.861849 -2.104569 -0.494929 1.071804 two
2013-01-05 -0.424972 0.567020 0.276232 -1.087401 four
3.5 增加新列
s1 = pd.Series([1, 2, 3, 4, 5, 6], index=pd.date_range("20130102", periods=6))
Out[46]:
2013-01-02 1
2013-01-03 2
2013-01-04 3
2013-01-05 4
2013-01-06 5
2013-01-07 6
Freq: D, dtype: int64
# 按轴索引添加数据
df["F"] = s1
不同方式设置值
# 通过标签设置值
df.at[dates[0], "A"] = 0
# 通过值位置设置值
df.iat[0, 1] = 0
# 指定 Numpy 数组为值
df.loc[:, "D"] = np.array([5] * len(df))
# 上述操作结果
Out[51]:
A B C D F
2013-01-01 0.000000 0.000000 -1.509059 5 NaN
2013-01-02 1.212112 -0.173215 0.119209 5 1.0
2013-01-03 -0.861849 -2.104569 -0.494929 5 2.0
2013-01-04 0.721555 -0.706771 -1.039575 5 3.0
2013-01-05 -0.424972 0.567020 0.276232 5 4.0
2013-01-06 -0.673690 0.113648 -1.478427 5 5.0
通过where操作后设置值
df2 = df.copy()
# 使所有值为负
df2[df2 > 0] = -df2
4 缺省值
pandas 使用 np.nan
代表缺省值
重索引允许更改/添加/删除指定轴上的索引,返回数据的副本
Out[51]:
A B C D F
2013-01-01 0.000000 0.000000 -1.509059 5 NaN
2013-01-02 1.212112 -0.173215 0.119209 5 1.0
2013-01-03 -0.861849 -2.104569 -0.494929 5 2.0
2013-01-04 0.721555 -0.706771 -1.039575 5 3.0
2013-01-05 -0.424972 0.567020 0.276232 5 4.0
2013-01-06 -0.673690 0.113648 -1.478427 5 5.0
df1 = df.reindex(index=dates[0:4], columns=list(df.columns) + ["E"])
df1.loc[dates[0] : dates[1], "E"] = 1
Out[57]:
A B C D F E
2013-01-01 0.000000 0.000000 -1.509059 5 NaN 1.0
2013-01-02 1.212112 -0.173215 0.119209 5 1.0 1.0
2013-01-03 -0.861849 -2.104569 -0.494929 5 2.0 NaN
2013-01-04 0.721555 -0.706771 -1.039575 5 3.0 NaN```
缺省值处理
```python
# 删除所有有缺省值的行
df1.dropna(how="any")
# 指定值填充缺省值
df1.fillna(value=5)
# 得到 boolean mask
pd.isna(df1)
Out[60]:
A B C D F E
2013-01-01 False False False False True False
2013-01-02 False False False False False False
2013-01-03 False False False False False True
2013-01-04 False False False False False True
4 操作
4.1 数据统计
此类操作会排除 nan 值
# 每列均值
df.mean()
# 指定轴
df.mean(1)
对于操作 pandas 会利用广播进行自动对齐
df = pd.DataFrame(np.random.randn(6, 4), index=pd.date_range("20130101", periods=6),columns=["A", "B", "C", "D"])
ds = pd.Series([1,2,3,4,5,6],index=pd.date_range("20130102", periods=6))
df["F"]=ds
dates=pd.date_range("20130101", periods=6)
s = pd.Series([1, 3, 5, np.nan, 6, 8], index=dates).shift(2)
Out[64]:
2013-01-01 NaN
2013-01-02 NaN
2013-01-03 1.0
2013-01-04 3.0
2013-01-05 5.0
2013-01-06 NaN
Freq: D, dtype: float64
df.sub(s, axis="index")
Out[65]:
A B C D F
2013-01-01 NaN NaN NaN NaN NaN
2013-01-02 NaN NaN NaN NaN NaN
2013-01-03 -1.861849 -3.104569 -1.494929 4.0 1.0
2013-01-04 -2.278445 -3.706771 -4.039575 2.0 0.0
2013-01-05 -5.424972 -4.432980 -4.723768 0.0 -1.0
2013-01-06 NaN NaN NaN NaN NaN
4.2 Apply 方法
df.apply(np.cumsum)
Out[66]:
A B C D F
2013-01-01 0.000000 0.000000 -1.509059 5 NaN
2013-01-02 1.212112 -0.173215 -1.389850 10 1.0
2013-01-03 0.350263 -2.277784 -1.884779 15 3.0
2013-01-04 1.071818 -2.984555 -2.924354 20 6.0
2013-01-05 0.646846 -2.417535 -2.648122 25 10.0
2013-01-06 -0.026844 -2.303886 -4.126549 30 15.0
df.apply(lambda x: x.max() - x.min())
Out[67]:
A 2.073961
B 2.671590
C 1.785291
D 0.000000
F 4.000000
dtype: float64
4.3 Histogramming
s = pd.Series(np.random.randint(0, 7, size=10))
s.value_counts()
Out[70]:
4 5
2 2
6 2
1 1
4.4 字符串方法
s = pd.Series(["A", "B", "C", "Aaba", "Baca", np.nan, "CABA", "dog", "cat"])
s.str.lower()
5 数据合并 Merge
5.1 Concat
df = pd.DataFrame(np.random.randn(10, 4))
# 将数据分为三部分
pieces = [df[:3], df[3:7], df[7:]]
# 合并数据,得到原数据
pd.concat(pieces)
值得注意的是,在 pandas 添加新列是非常快的,而添加行需要用到复制操作,代价相对较高
5.2 Join
SQL 方式的合并
left = pd.DataFrame({"key": ["foo", "bar"], "lval": [1, 2]})
right = pd.DataFrame({"key": ["foo", "bar"], "rval": [4, 5]})
pd.merge(left, right, on="key")
Out[86]:
key lval rval
0 foo 1 4
1 bar 2 5
6 分组 Grouping
分组一般涉及以下一个或多个步骤的流程:
- Splitting :根据某一准则将数据分裂
- Applying :对每一组应用某个方法
- Combining:将结果数据进行关联
详见添加链接描述
df = pd.DataFrame(
{
"A": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],
"B": ["one", "one", "two", "three", "two", "two", "one", "three"],
"C": np.random.randn(8),
"D": np.random.randn(8),
}
)
# 按某列分组,然后应用方法、
df.groupby("A").sum()
Out[89]:
C D
A
bar 1.732707 1.073134
foo 2.824590 -0.574779
# 多列分组
df.groupby(["A", "B"]).sum()
Out[90]:
C D
A B
bar one 1.511763 0.396823
three -0.990582 -0.532532
two 1.211526 1.208843
foo one 1.614581 -1.658537
three 0.024580 -0.264610
two 1.185429 1.348368
7 Reshaping
待补充
7.1 Stack
7.2Pivot tables
8 时间序列
待补充
9 Categoricals
待补充
10 Plotting
待补充
11 载入数据与输出数据
11.1 CSV
# 输出到 csv
df.to_csv("foo.csv")
# 从 csv 读取数据
pd.read_csv("foo.csv")
11.2 HDF5
df.to_hdf("foo.h5", "df")
pd.read_hdf("foo.h5", "df")
11.3 Excel
df.to_excel("foo.xlsx", sheet_name="Sheet1")
pd.read_excel("foo.xlsx", "Sheet1", index_col=None, na_values=["NA"])