目录
Pandas
Pandas是Python 中的数据操纵和分析软件包。名称“Pandas”得名自计量经济学 Panel Data（面板数据）一词。
Pandas为Python带来了两个新的数据结构，即Pandas Series和Pandas DataFrame。借助这两个数据结构，我们能够轻松直观地处理带标签数据和关系数据。
机器学习的重要一步是首先检查数据，通过进行一些基本的数据分析，确保数据很适合你的训练算法。对于数据来说，数量并不是唯一重要的方面，数据质量也同等重要。经常大型数据库并不能直接馈送到学习算法中，很多时候存在大型数据集缺失值、离群值、不正确的值，等等。这时候，Pandas 就派上用场了。Pandas Series 和 DataFrame 专门用于快速进行数据分析和操纵。
Series
A Pandas Series is a one-dimensional array-like object that can hold many data types and you can assign an index label to each element in the Pandas Series.
带索引的一维数组，有两个属性，values和index。
1. 默认情况下，创建一个Series时，会自动添加数字形式的索引
import pandas as pd
s = pd.Series(["a","b","c"])
print(s)
print(type(s))
print(s.values, s.index)
print(s.ndim, s.shape, s.size)
0 a
1 b
2 c
dtype: object
<class 'pandas.core.series.Series'>
['a' 'b' 'c'] RangeIndex(start=0, stop=3, step=1)
1 (3,) 3
[Finished in 0.5s]2. 可以加入index参数来指定索引
import pandas as pd
import numpy as np
s = pd.Series(["a","b","c"], index=[0,1,2])
print(s)
print(type(s))
print(s.values, s.index)
print(s.ndim, s.shape, s.size)
print("*********")
s = pd.Series(["a","b","c"], index=[1,2,3])
print(s)
print(type(s))
print(s.values, s.index)
print(s.ndim, s.shape, s.size)
print("*********")
s = pd.Series(np.arange(3), index=["a","b","c"])
print(s)
print(type(s))
print(s.values, s.index)
print(s.ndim, s.shape, s.size)
0 a
1 b
2 c
dtype: object
<class 'pandas.core.series.Series'>
['a' 'b' 'c'] Int64Index([0, 1, 2], dtype='int64')
1 (3,) 3
*********
1 a
2 b
3 c
dtype: object
<class 'pandas.core.series.Series'>
['a' 'b' 'c'] Int64Index([1, 2, 3], dtype='int64')
1 (3,) 3
*********
a 0
b 1
c 2
dtype: int64
<class 'pandas.core.series.Series'>
[0 1 2] Index(['a', 'b', 'c'], dtype='object')
1 (3,) 3
[Finished in 0.4s]3. 可以使用字典创建Series：
如果没有使用index参数，Series的index - 字典的key，Series的values - 字典的value；
如果使用了index参数，Series的index - 以index参数为准，如果字典中没有该key，values为NaN(Not a Number)
import pandas as pd
data = {"test1score":80, "test2score":70}
print(data, "\n")
s = pd.Series(data)
print(s)
print(type(s))
print(s.values, s.index)
print(s.ndim, s.shape, s.size)
print()
s = pd.Series(data, index=["test1score", "test3score"])
print(s)
print(type(s))
print(s.values, s.index)
print(s.ndim, s.shape, s.size)
{'test1score': 80, 'test2score': 70}
test1score 80
test2score 70
dtype: int64
<class 'pandas.core.series.Series'>
[80 70] Index(['test1score', 'test2score'], dtype='object')
1 (2,) 2
test1score 80.0
test3score NaN
dtype: float64
<class 'pandas.core.series.Series'>
[80. nan] Index(['test1score', 'test3score'], dtype='object')
1 (2,) 2
[Finished in 0.4s]eg.
import pandas as pd
# We create a Pandas Series that stores a grocery list
groceries = pd.Series(data=[30, 6, 'Yes', 'No'], index=['eggs', 'apples', 'milk', 'bread'])
print(groceries)
x1 = 'bananas' in groceries.index
x2 = 'bananas' in groceries
print('Is bananas an index label in Groceries:', x1)
print('Is bananas an index label in Groceries:', x2)
y1 = 'bread' in groceries.index
y2 = 'bread' in groceries
print('Is bread an index label in Groceries:', y1)
print('Is bread an index label in Groceries:', y2)
u = 'No' in groceries.values
print("Is 'No' a value in Groceries:", u)
v = 60 in groceries.values
print('Is 60 a value in Groceries:', v)
eggs 30
apples 6
milk Yes
bread No
dtype: object
Is bananas an index label in Groceries: False
Is bananas an index label in Groceries: False
Is bread an index label in Groceries: True
Is bread an index label in Groceries: True
Is 'No' a value in Groceries: True
Is 60 a value in Groceries: False
[Finished in 0.4s]索引、切片
索引
1. access elements using index labels/numerical indices
在方括号[]内添加索引标签或数字索引访问元素。使用数字索引时，使用正整数从Series的开头访问数据，或使用负整数从末尾访问。
Pandas Series 提供了两个属性 .loc 和 .iloc。属性 .loc 表示位置，用于明确表明我们使用的是标签索引。\属性.iloc 表示整型位置，用于明确表明我们使用的是数字索引。
2. 通过索引获取多个值
import pandas as pd
groceries = pd.Series(data=[30, 6, 'Yes', 'No'], index=['eggs', 'apples', 'milk', 'bread'])
print(groceries)
print()
print('How many eggs do we need to buy:', groceries['eggs'])
print('How many eggs do we need to buy:', groceries.loc['eggs'])
print('How many eggs do we need to buy:', groceries[0])
print('How many eggs do we need to buy:', groceries.iloc[0])
print()
print('Do we need milk and bread:', groceries[['milk','bread']].values)
print('Do we need milk and bread:', groceries.loc[['milk','bread']].values)
print('Do we need milk and bread:', groceries[[-2,-1]].values)
print('Do we need milk and bread:', groceries.iloc[[-2,-1]].values)
print('Do we need milk and bread:\n{}'.format(groceries[['milk','bread']]))eggs 30
apples 6
milk Yes
bread No
dtype: object
How many eggs do we need to buy: 30
How many eggs do we need to buy: 30
How many eggs do we need to buy: 30
How many eggs do we need to buy: 30
Do we need milk and bread: ['Yes' 'No']
Do we need milk and bread: ['Yes' 'No']
Do we need milk and bread: ['Yes' 'No']
Do we need milk and bread: ['Yes' 'No']
Do we need milk and bread:
milk Yes
bread No
dtype: object
[Finished in 0.5s]3. 通过索引对Series的values进行修改
4. （基于2、3）通过多个索引进行赋值操作
import pandas as pd
s = pd.Series(data=[1,2,3,4,5], index=["a","b","c","d","e"])
print(s)
print(s["a"])
s["a"] = 9
print(s)
print(s[["b","d","e"]])
s[["b","d","e"]] = [s["d"],s["e"],s["b"]]
print(s)
s[["b","d","e"]] = 0
print(s)
print("*********")
data = {"test1score":80, "test2score":70}
s = pd.Series(data, index=["test1score","test2score","test3score"])
print(s)
print(s[["test1score","test3score"]])
a 1
b 2
c 3
d 4
e 5
dtype: int64
1
a 9
b 2
c 3
d 4
e 5
dtype: int64
b 2
d 4
e 5
dtype: int64
a 9
b 4
c 3
d 5
e 2
dtype: int64
a 9
b 0
c 3
d 0
e 0
dtype: int64
*********
test1score 80.0
test2score 70.0
test3score NaN
dtype: float64
test1score 80.0
test3score NaN
dtype: float64
[Finished in 0.5s]布尔索引
pandas.isnull()获取Series中values为空的值，pandas.notnull()获取Series中values不为空的值
import pandas as pd
s = pd.Series(data=[1,2,3,4,5], index=["a","b","c","d","e"])
print(s)
print(s>2)
print(s[s>2])
s[s>2] = -1
print(s)
print("*********")
data = {"test1score":80, "test2score":70}
s = pd.Series(data, index=["test1score","test2score","test3score"])
print(s)
print(pd.isnull(s))
print(s[pd.isnull(s)])
print(s[pd.notnull(s)])
s[pd.isnull(s)] = 0
print(s)a 1
b 2
c 3
d 4
e 5
dtype: int64
a False
b False
c True
d True
e True
dtype: bool
c 3
d 4
e 5
dtype: int64
a 1
b 2
c -1
d -1
e -1
dtype: int64
*********
test1score 80.0
test2score 70.0
test3score NaN
dtype: float64
test1score False
test2score False
test3score True
dtype: bool
test3score NaN
dtype: float64
test1score 80.0
test2score 70.0
dtype: float64
test1score 80.0
test2score 70.0
test3score 0.0
dtype: float64
[Finished in 0.4s]切片
1. = 为引用传递，并不是值传递
2. copy函数，在堆中再创建内容相同的一份
import pandas as pd
s = pd.Series([1,2,3,4,5], index=["a","b","c","d","e"])
print('s:\n{}'.format(s))
s2 = s[1:4]
print('s2:\n{}'.format(s2))
s2[0:2] = [9,8]
print('s2:\n{}'.format(s2))
print('s:\n{}'.format(s))
print()
s = pd.Series([1,2,3,4,5], index=["a","b","c","d","e"])
print('s:\n{}'.format(s))
s3 = s[1:4].copy()
print('s3:\n{}'.format(s3))
s3[0:2] = [9,8]
print('s3:\n{}'.format(s3))
print('s:\n{}'.format(s))
s:
a 1
b 2
c 3
d 4
e 5
dtype: int64
s2:
b 2
c 3
d 4
dtype: int64
s2:
b 9
c 8
d 4
dtype: int64
s:
a 1
b 9
c 8
d 4
e 5
dtype: int64
s:
a 1
b 2
c 3
d 4
e 5
dtype: int64
s3:
b 2
c 3
d 4
dtype: int64
s3:
b 9
c 8
d 4
dtype: int64
s:
a 1
b 2
c 3
d 4
e 5
dtype: int64
[Finished in 0.5s]
操作
删除
Series.drop(label)：从给定 Series 中删除给定的 label，不在原地地从 Series 中删除元素，即不会更改被修改的原始 Series。
Series.drop(label, inplace=True)：从给定 Series 中删除给定的 label，原地地从 Pandas Series 中删除条目。
import pandas as pd
# We create a Pandas Series that stores a grocery list
groceries = pd.Series(data=[30, 6, 'Yes', 'No'], index=['eggs', 'apples', 'milk', 'bread'])
print('Original Grocery List:\n{}'.format(groceries))
# We remove apples from our grocery list. The drop function removes elements out of place
print("We remove apples by groceries.drop('apples'):\n{}".format(groceries.drop('apples')))
# When we remove elements out of place the original Series remains intact.
print('Modified Grocery List:\n{}'.format(groceries))
# We remove apples from our grocery list in place by setting the inplace keyword to True
print("We remove apples by groceries.drop('apples', inplace=True):\n{}".format(groceries.drop('apples', inplace=True)))
print('Modified Grocery List:\n{}'.format(groceries))
Original Grocery List:
eggs 30
apples 6
milk Yes
bread No
dtype: object
We remove apples by groceries.drop('apples'):
eggs 30
milk Yes
bread No
dtype: object
Modified Grocery List:
eggs 30
apples 6
milk Yes
bread No
dtype: object
We remove apples by groceries.drop('apples', inplace=True):
None
Modified Grocery List:
eggs 30
milk Yes
bread No
dtype: object
[Finished in 0.5s]
计算
Series和矢量（数值）的运算
Series的每一个values和数值进行计算
import pandas as pd
# We create a Pandas Series that stores a grocery list of just fruits
fruits = pd.Series(data=[10,6,3], index=['apples','oranges','bananas'])
print('Original grocery list of fruits:\n{}'.format(fruits))
# We perform basic element-wise operations using arithmetic symbols
print('\nfruits + 10:\n{}'.format(fruits + 10)) # We add 10 to each item in fruits
print('\nfruits - 10:\n{}'.format(fruits - 10)) # We subtract 10 to each item in fruits
print('\nfruits * 10:\n{}'.format(fruits * 10)) # We multiply each item in fruits by 10
print('\nfruits / 10:\n{}'.format(fruits / 10)) # We divide each item in fruits by 10
Original grocery list of fruits:
apples 10
oranges 6
bananas 3
dtype: int64
fruits + 10:
apples 20
oranges 16
bananas 13
dtype: int64
fruits - 10:
apples 0
oranges -4
bananas -7
dtype: int64
fruits * 10:
apples 100
oranges 60
bananas 30
dtype: int64
fruits / 10:
apples 1.0
oranges 0.6
bananas 0.3
dtype: float64
[Finished in 0.4s]import pandas as pd
# We create a Pandas Series that stores a grocery list
groceries = pd.Series(data=[30, 6, 'Yes', 'No'], index=['eggs', 'apples', 'milk', 'bread'])
print('Original Grocery List:\n{}'.format(groceries))
# We multiply our grocery list by 2
print('\nWe multiply our grocery list by 2:\n{}'.format(groceries * 2))
Original Grocery List:
eggs 30
apples 6
milk Yes
bread No
dtype: object
We multiply our grocery list by 2:
eggs 60
apples 12
milk YesYes
bread NoNo
dtype: object
[Finished in 0.4s]Series和Series的运算
前提是要有相同的index，对应index的values进行计算
import pandas as pd
data1 = {"Jack": 92, "James": 59, "Jane": 60, "Jennifer": 70, "Jill": 80}
data2 = {"Jack": 79, "James": 64, "Jane": 75, "Janet": 50, "Jill": 83}
print(data1)
print(data2)
print()
s1 = pd.Series(data1)
s2 = pd.Series(data2)
print(s1)
print(s2)
print(s1 + s2)
print()
s1 = pd.Series(data1, index=["Jane", "Jill", "Jack", "James"])
s2 = pd.Series(data2, index=["Jill", "Jane", "James", "Jack"])
print(s1)
print(s2)
print(s1 + s2)
{'Jack': 92, 'James': 59, 'Jane': 60, 'Jennifer': 70, 'Jill': 80}
{'Jack': 79, 'James': 64, 'Jane': 75, 'Janet': 50, 'Jill': 83}
Jack 92
James 59
Jane 60
Jennifer 70
Jill 80
dtype: int64
Jack 79
James 64
Jane 75
Janet 50
Jill 83
dtype: int64
Jack 171.0
James 123.0
Jane 135.0
Janet NaN
Jennifer NaN
Jill 163.0
dtype: float64
Jane 60
Jill 80
Jack 92
James 59
dtype: int64
Jill 83
Jane 75
James 64
Jack 79
dtype: int64
Jack 171
James 123
Jane 135
Jill 163
dtype: int64
[Finished in 0.5s]
数学函数
import pandas as pd
# We create a Pandas Series that stores a grocery list of just fruits
fruits = pd.Series(data=[10,6,3], index=['apples','oranges','bananas'])
print('Original grocery list of fruits:\n{}'.format(fruits))
# We import NumPy as np to be able to use the mathematical functions
import numpy as np
# We apply different mathematical functions to all elements of fruits
print('\nEXP(X) = \n{}'.format(np.exp(fruits)))
print('\nSQRT(X) =\n{}'.format(np.sqrt(fruits)))
print('\nPOWER(X,2) =\n{}'.format(np.power(fruits,2)))
Original grocery list of fruits:
apples 10
oranges 6
bananas 3
dtype: int64
EXP(X) =
apples 22026.465795
oranges 403.428793
bananas 20.085537
dtype: float64
SQRT(X) =
apples 3.162278
oranges 2.449490
bananas 1.732051
dtype: float64
POWER(X,2) =
apples 100
oranges 36
bananas 9
dtype: int64
[Finished in 0.5s]
DataFrame
DataFrame is a two-dimensional object with labeled rows and columns and can also hold multiple data types.
二维表，往往使用SQL操作数据库达不到想要的结果时，才使用Python中的DataFrame。
结构化数据：excel表格（也是二维表），数据库的table
非结构化数据：文本、图片、视频
半结构化数据：xml文本
columns获取列名，info()获取每一列的信息，shape获取结构。
可以通过columns和index参数来指定列名和索引，如果columns参数指定的列在字典中没有匹配项，那么会保留该列名，值为NaN。
1. 通过a Dictionary创建Pandas DataFrame
1.1 通过a Dictionary of Pandas Series创建Pandas DataFrame
最低0.47元/天 解锁文章
128
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
