本篇博客主要内容
1)移除重复数据(duplicated)
2)利用函数或映射进行数据转换(map)
3)替换值(replace)
4)重命名轴索引
5)检测和过滤异常值(逻辑索引)
6)随机采样或选出随机子集
7)计算哑变量(get_dummies)
1)移除重复数据
检查某列数据是否重复可用.is_unique
检查某行数据是否重复可用.duplicated
import pandas as pd
import numpy as np
data = pd.DataFrame({
'k1':['one'] * 3 + ['two'] * 4,'k2':[1,1,2,3,3,4,4]})
data
k1 | k2 | |
---|---|---|
0 | one | 1 |
1 | one | 1 |
2 | one | 2 |
3 | two | 3 |
4 | two | 3 |
5 | two | 4 |
6 | two | 4 |
检查列以及行中重复数据
data.index.is_unique#检查列
True
data.k1.is_unique#检查列
False
data['k2'].is_unique#检查列
False
data.is_unique
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-12-3c5fb82b7563> in <module>()
----> 1 data.is_unique
C:\Program Files\anaconda\lib\site-packages\pandas\core\generic.pyc in __getattr__(self, name)
2670 if name in self._info_axis:
2671 return self[name]
-> 2672 return object.__getattribute__(self, name)
2673
2674 def __setattr__(self, name, value):
AttributeError: 'DataFrame' object has no attribute 'is_unique'
data.duplicated()#检查行
0 False
1 True
2 False
3 False
4 True
5 False
6 True
dtype: bool
移除重复行
data.drop_duplicates()
k1 | k2 | |
---|---|---|
0 | one | 1 |
2 | one | 2 |
3 | two | 3 |
5 | two | 4 |
set(data.k1)#保留唯一的列属性值
{
'one', 'two'}
移除重复值小结
1) drop_duplicates、duolicated函数只能用于DataFrame
2) is_unique不能用于DataFrame
2)利用函数或映射进行数据转换
data1 = pd.DataFrame({
'food':['bacon','pork','bacon','Pastrami',\
'beef','Bacon','pastrami','ham','lox'],\
'ounces':[4,3,12,6,7.5,8,3,5,6]})
data1
food | ounces | |
---|---|---|
0 | bacon | 4.0 |
1 | pork | 3.0 |
2 | bacon | 12.0 |
3 | Pastrami | 6.0 |
4 | beef | 7.5 |
5 | Bacon | 8.0 |
6 | pastrami | 3.0 |
7 | ham | 5.0 |
8 | lox | 6.0 |
添加一列表示肉类来源的动物类型
#step1:构建肉类到动物的映射
meat_to_animal = {
'bacon':'pig','pork':'pig','pastrami':'cow',