闲话笔记之运用pandas处理缺失值

最新推荐文章于 2022-05-13 20:17:28 发布

iamwxy123

最新推荐文章于 2022-05-13 20:17:28 发布

阅读量330

点赞数

文章标签： python 数据分析

本文链接：https://blog.csdn.net/iamwxy123/article/details/105966897

版权

文章目录

1.如何检测缺失值: df.isna() 和 df.notna()

In [6]: df2
Out[6]: 
        one       two     three four   five
a  0.469112 -0.282863 -1.509059  bar   True
b       NaN       NaN       NaN  NaN    NaN
c -1.135632  1.212112 -0.173215  bar  False
d       NaN       NaN       NaN  NaN    NaN
e  0.119209 -1.044236 -0.861849  bar   True
f -2.104569 -0.494929  1.071804  bar  False
g       NaN       NaN       NaN  NaN    NaN
h  0.721555 -0.706771 -1.039575  bar   True

In [7]:import pandas as pd
In [8]: pd.isna(df2['one']) #与df2['one'].isna()等同,但如果使用df2['one']==np.nan,返回值全部为False.
Out[8]: 
a    False
b     True
c    False
d     True
e    False
f    False
g     True
h    False
Name: one, dtype: bool

In [9]: df2['four'].notna()
Out[9]: 
a     True
b    False
c     True
d    False
e     True
f     True
g    False
h     True
Name: four, dtype: bool

2.如何创建含空值的int型数列

In[52]:pd.Series([1, 2, np.nan, 4])#np.nan是一个浮点数,含有空值的int型数组会被强制转换成float型数组
Out[52]: 
0    1.0
1    2.0
2    NaN
3    4.0
dtype: float64

In[51]:pd.Series([1, 2, np.nan, 4], dtype=pd.Int64Dtype()) #创建可为空值的int型数组
Out[51]: 
0      1
1      2
2    NaN
3      4
dtype: Int64

3.日期格式的缺失标记是NaT

In [17]: df2
Out[17]: 
        one       two     three four   five  timestamp
a  0.469112 -0.282863 -1.509059  bar   True 2012-01-01
c -1.135632  1.212112 -0.173215  bar  False 2012-01-01
e  0.119209 -1.044236 -0.861849  bar   True 2012-01-01
f -2.104569 -0.494929  1.071804  bar  False 2012-01-01
h  0.721555 -0.706771 -1.039575  bar   True 2012-01-01

In [18]: df2.loc[['a', 'c', 'h'], ['one', 'timestamp']] = np.nan

In [19]: df2
Out[19]: 
        one       two     three four   five  timestamp
a       NaN -0.282863 -1.509059  bar   True        NaT
c       NaN  1.212112 -0.173215  bar  False        NaT
e  0.119209 -1.044236 -0.861849  bar   True 2012-01-01
f -2.104569 -0.494929  1.071804  bar  False 2012-01-01
h       NaN -0.706771 -1.039575  bar   True        NaT

4.插入缺失值

对于数值型字段,pandas总是使用NaN作为缺失值的标记;
对于日期型字段,pandas总是使用NaT作为缺失值的标记;
对于字符串型字段,pandas使用给定的值作为缺失值的标记.

In [21]: s = pd.Series([1, 2, 3]) #数值型
In [22]: s.loc[0] = None #缺失值显示仍为NaN
In [23]: s
Out[23]: 
0    NaN
1    2.0
2    3.0
dtype: float64

In [24]: s = pd.Series(["a", "b", "c"]) #字符串型
In [25]: s.loc[0] = None
In [26]: s.loc[1] = np.nan
In [27]: s
Out[27]: 
0    None
1     NaN
2       c
dtype: object

5.在算术运算中,含有缺失值的字段如何参与计算

一般是忽略空值,然后进行计算.

In [31]: df
Out[31]: 
        one       two     three
a       NaN -0.282863 -1.509059
c       NaN  1.212112 -0.173215
e  0.119209 -1.044236 -0.861849
f -2.104569 -0.494929  1.071804
h       NaN -0.706771 -1.039575

In [32]: df['one'].sum()  #忽略空值
Out[32]: -1.9853605075978744

In [33]: df.mean(1)   #忽略空值
Out[33]: 
a   -0.895961
c    0.519449
e   -0.595625
f   -0.509232
h   -0.873173
dtype: float64

In [34]: df.cumsum()  #忽略空值
Out[34]: 
        one       two     three
a       NaN -0.282863 -1.509059
c       NaN  0.929249 -1.682273
e  0.119209 -0.114987 -2.544122
f -1.985361 -0.609917 -1.472318
h       NaN -1.316688 -2.511893

In [35]: df.cumsum(skipna=False) #体现空值,不忽略
Out[35]: 
   one       two     three
a  NaN -0.282863 -1.509059
c  NaN  0.929249 -1.682273
e  NaN -0.114987 -2.544122
f  NaN -0.609917 -1.472318
h  NaN -1.316688 -2.511893

6.所有空值的总和等于0,所有空值的连乘等于1

In [36]: pd.Series([np.nan]).sum()
Out[36]: 0.0
In [37]: pd.Series([]).sum()
Out[37]: 0.0

In [38]: pd.Series([np.nan]).prod()
Out[38]: 1.0
In [39]: pd.Series([]).prod()
Out[39]: 1.0

7.分组运算中直接排除NaN这一组

In [40]: df
Out[40]: 
        one       two     three
a       NaN -0.282863 -1.509059
c       NaN  1.212112 -0.173215
e  0.119209 -1.044236 -0.861849
f -2.104569 -0.494929  1.071804
h       NaN -0.706771 -1.039575

In [41]: df.groupby('one').mean()
Out[41]: 
                two     three
one                          
-2.104569 -0.494929  1.071804
 0.119209 -1.044236 -0.861849

8.对空值进行填充:fillna()

(1)用指定值进行填充

In [42]: df2
Out[42]: 
        one       two     three four   five  timestamp
a       NaN -0.282863 -1.509059  bar   True        NaT
c       NaN  1.212112 -0.173215  bar  False        NaT
e  0.119209 -1.044236 -0.861849  bar   True 2012-01-01
f -2.104569 -0.494929  1.071804  bar  False 2012-01-01
h       NaN -0.706771 -1.039575  bar   True        NaT

In [43]: df2.fillna(0) #对整个dataframe的空值进行填充
Out[43]: 
        one       two     three four   five            timestamp
a  0.000000 -0.282863 -1.509059  bar   True                    0
c  0.000000  1.212112 -0.173215  bar  False                    0
e  0.119209 -1.044236 -0.861849  bar   True  2012-01-01 00:00:00
f -2.104569 -0.494929  1.071804  bar  False  2012-01-01 00:00:00
h  0.000000 -0.706771 -1.039575  bar   True                    0

In [44]: df2['one'].fillna('missing') #对指定列的空值进行填充
Out[44]: 
a     missing
c     missing
e    0.119209
f    -2.10457
h     missing
Name: one, dtype: object

(2)用所在列的前一个(或后一个)非空值进行填充

In [45]: df
Out[45]: 
        one       two     three
a       NaN -0.282863 -1.509059
c       NaN  1.212112 -0.173215
e  0.119209 -1.044236 -0.861849
f -2.104569 -0.494929  1.071804
h       NaN -0.706771 -1.039575

In [46]: df.fillna(method='pad')#用前一个非空值进行填充
Out[46]: 
        one       two     three
a       NaN -0.282863 -1.509059
c       NaN  1.212112 -0.173215
e  0.119209 -1.044236 -0.861849
f -2.104569 -0.494929  1.071804
h -2.104569 -0.706771 -1.039575

In [47]: df
Out[47]: 
   one       two     three
a  NaN -0.282863 -1.509059
c  NaN  1.212112 -0.173215
e  NaN       NaN       NaN
f  NaN       NaN       NaN
h  NaN -0.706771 -1.039575

In [48]: df.fillna(method='pad', limit=1)#limit指定连续空值中被填充的个数
Out[48]: 
   one       two     three
a  NaN -0.282863 -1.509059
c  NaN  1.212112 -0.173215
e  NaN  1.212112 -0.173215
f  NaN       NaN       NaN
h  NaN -0.706771 -1.039575

(3)用所在列的平均值进行填充

In [53]: dff
Out[53]: 
          A         B         C
0  0.271860 -0.424972  0.567020
1  0.276232 -1.087401 -0.673690
2  0.113648 -1.478427  0.524988
3       NaN  0.577046 -1.715002
4       NaN       NaN -1.157892
5 -1.344312       NaN       NaN
6 -0.109050  1.643563       NaN
7  0.357021 -0.674600       NaN
8 -0.968914 -1.294524  0.413738
9  0.276662 -0.472035 -0.013960

In [54]: dff.fillna(dff.mean())
Out[54]: 
          A         B         C
0  0.271860 -0.424972  0.567020
1  0.276232 -1.087401 -0.673690
2  0.113648 -1.478427  0.524988
3 -0.140857  0.577046 -1.715002
4 -0.140857 -0.401419 -1.157892
5 -1.344312 -0.401419 -0.293543
6 -0.109050  1.643563 -0.293543
7  0.357021 -0.674600 -0.293543
8 -0.968914 -1.294524  0.413738
9  0.276662 -0.472035 -0.013960

In [55]: dff.fillna(dff.mean()['B':'C'])
Out[55]: 
          A         B         C
0  0.271860 -0.424972  0.567020
1  0.276232 -1.087401 -0.673690
2  0.113648 -1.478427  0.524988
3       NaN  0.577046 -1.715002
4       NaN -0.401419 -1.157892
5 -1.344312 -0.401419 -0.293543
6 -0.109050  1.643563 -0.293543
7  0.357021 -0.674600 -0.293543
8 -0.968914 -1.294524  0.413738
9  0.276662 -0.472035 -0.013960

(4)直接删除含空值的行或列

df.dropna(axis=0)  #删除有空值的行
df.dropna(axis=1)  #删除有空值的列

(5)线性插值:interpolation()

In [70]: ser
Out[70]: 
0.0      0.0
1.0      NaN
10.0    10.0
dtype: float64

In [71]: ser.interpolate()#忽略索引的插值法
Out[71]: 
0.0      0.0
1.0      5.0
10.0    10.0
dtype: float64

In [72]: ser.interpolate(method='values')#参考索引的插值法,method还可以指定更复杂的插值方法
Out[72]: 
0.0      0.0
1.0      1.0
10.0    10.0
dtype: float64

#另外一个例子
In [73]: df = pd.DataFrame({'A': [1, 2.1, np.nan, 4.7, 5.6, 6.8],
   ....:                    'B': [.25, np.nan, np.nan, 4, 12.2, 14.4]})

In [74]: df
Out[74]: 
     A      B
0  1.0   0.25
1  2.1    NaN
2  NaN    NaN
3  4.7   4.00
4  5.6  12.20
5  6.8  14.40

In [75]: df.interpolate()#默认是忽略索引的线性插值法
Out[75]: 
     A      B
0  1.0   0.25
1  2.1   1.50
2  3.4   2.75
3  4.7   4.00
4  5.6  12.20
5  6.8  14.40

#其他插值方法
In [76]: df.interpolate(method='barycentric')
Out[76]: 
      A       B
0  1.00   0.250
1  2.10  -7.660
2  3.53  -4.515
3  4.70   4.000
4  5.60  12.200
5  6.80  14.400

In [77]: df.interpolate(method='pchip')
Out[77]: 
         A          B
0  1.00000   0.250000
1  2.10000   0.672808
2  3.43454   1.928950
3  4.70000   4.000000
4  5.60000  12.200000
5  6.80000  14.400000

In [78]: df.interpolate(method='akima')
Out[78]: 
          A          B
0  1.000000   0.250000
1  2.100000  -0.873316
2  3.406667   0.320034
3  4.700000   4.000000
4  5.600000  12.200000
5  6.800000  14.400000

In [128]: df.replace(1.5, np.nan, inplace=True) #加上inplace=True表示修改了原来的df

iamwxy123

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
闲话笔记之运用pandas处理缺失值

文章目录1.如何检测缺失值: df.isna() 和 df.notna()2.如何创建含空值的int型数列3.日期格式的缺失标记是NaT4.插入缺失值5.在算术运算中,含有缺失值的字段如何参与计算6.所有空值的总和等于0,所有空值的连乘等于17.分组运算中直接排除NaN这一组8.对空值进行填充:fillna()1.如何检测缺失值: df.isna() 和 df.notna()In [6]:...
复制链接

扫一扫