数据分析处理库Pandas-常用函数

导入pandas库和numpy库

import pandas as pd
import numpy as np

我们以一个csv文件来展示pandas是如何来进行数据预处理的：titanic_train.csv

读入文件titanic_train.csv

titanic_survival = pd.read_csv("titanic_train.csv")

1、求平均值
①通过求和和求长函数来计算平均值

#去除缺失值
age_is_null = pd.isnull(titanic_survival["Age"])
good_ages = titanic_survival["Age"][age_is_null == False]
#求和后的数据除以总数据长度
correct_mean_age = sum(good_ages) / len(good_ages)
print(correct_mean_age)

OUT：

29.6991176471

②通过pandas函数来求平均值

correct_mean_age_pn = titanic_survival["Age"].mean()
print(correct_mean_age_pn)

OUT:

29.69911764705882

③计算不同等级船舱对应的平均消费

passenger_classes = [1, 2, 3]
fares_by_class = {}
for this_class in passenger_classes:
    #取出Pclass值为this_class的行
    pclass_rows = titanic_survival[titanic_survival["Pclass"] == this_class]
    #取出Pclass值为this_class的行的Fare值
    pclass_fare = pclass_rows["Fare"]
    #计算取出的Fare值的平均数
    fare_to_class = pclass_fare.mean()
    #将计算出的平均数传入fares_by_class字典
    fares_by_class[this_class] = fare_to_class
print(fares_by_class)

OUT:

{1: 84.15468749999992, 2: 20.66218315217391, 3: 13.675550101832997}

2、透视表
①统计船舱等级和获救几率之间的关系

passenger_survival = titanic_survival.pivot_table(index="Pclass", values="Survived", aggfunc=np.mean)
print(passenger_survival)

参数解释：
index:统计基准
values:被统计数据
aggfunc:统计方式(默认值是mean)

OUT:

Pclass  Survived        
1       0.629630
2       0.472826
3       0.242363

②统计性别和获救几率的关系

sex_survival = titanic_survival.pivot_table(index="Sex", values="Survived", aggfunc=np.mean)
print(sex_survival)

OUT：

Sex     Survived        
female  0.742038
male    0.188908

③统计登船地点与消费和获救人数的关系

port_stats = titanic_survival.pivot_table(index="Embarked", values=["Fare", "Survived"], aggfunc=np.sum)
print(port_stats)

OUT:

Embarked     Fare        Survived        
C         10072.2962        93
Q          1022.2543        30
S         17439.3988       217

3、去除缺失值
①去除有缺失值的列

drop_na_columns = titanic_survival.dropna(axis=1)

OUT：

   PassengerId  Survived  Pclass  \
0            1         0       3   
1            2         1       1   
2            3         1       3   
3            4         1       1   
4            5         0       3   
5            6         0       3   
6            7         0       1   
7            8         0       3   
8            9         1       3   
9           10         1       2   

                                                Name     Sex  SibSp  Parch  \
0                            Braund, Mr. Owen Harris    male      1      0   
1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female      1      0   
2                             Heikkinen, Miss. Laina  female      0      0   
3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female      1      0   
4                           Allen, Mr. William Henry    male      0      0   
5                                   Moran, Mr. James    male      0      0   
6                            McCarthy, Mr. Timothy J    male      0      0   
7                     Palsson, Master. Gosta Leonard    male      3      1   
8  Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)  female      0      2   
9                Nasser, Mrs. Nicholas (Adele Achem)  female      1      0   

             Ticket     Fare  
0         A/5 21171   7.2500  
1          PC 17599  71.2833  
2  STON/O2. 3101282   7.9250  
3            113803  53.1000  
4            373450   8.0500  
5            330877   8.4583  
6             17463  51.8625  
7            349909  21.0750  
8            347742  11.1333  
9            237736  30.0708 

②去除有缺失值的行

new_titanic_survival = titanic_survival.dropna(axis=0, subset=["Age", "Sex"])
print(new_titanic_survival)

OUT：

PassengerId  Survived  Pclass  \
0             1         0       3   
1             2         1       1   
2             3         1       3   
3             4         1       1   
4             5         0       3   
6             7         0       1   
7             8         0       3   
8             9         1       3   
9            10         1       2   
10           11         1       3   

                                                 Name     Sex   Age  SibSp  \
0                             Braund, Mr. Owen Harris    male  22.0      1   
1   Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   
2                              Heikkinen, Miss. Laina  female  26.0      0   
3        Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   
4                            Allen, Mr. William Henry    male  35.0      0   
6                             McCarthy, Mr. Timothy J    male  54.0      0   
7                      Palsson, Master. Gosta Leonard    male   2.0      3   
8   Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)  female  27.0      0   
9                 Nasser, Mrs. Nicholas (Adele Achem)  female  14.0      1   
10                    Sandstrom, Miss. Marguerite Rut  female   4.0      1   

    Parch            Ticket     Fare Cabin Embarked  
0       0         A/5 21171   7.2500   NaN        S  
1       0          PC 17599  71.2833   C85        C  
2       0  STON/O2. 3101282   7.9250   NaN        S  
3       0            113803  53.1000  C123        S  
4       0            373450   8.0500   NaN        S  
6       0             17463  51.8625   E46        S  
7       1            349909  21.0750   NaN        S  
8       2            347742  11.1333   NaN        S  
9       0            237736  30.0708   NaN        C  
10      1           PP 9549  16.7000    G6        S  

4、定位

row_index_age = titanic_survival.loc[83, "Age"]
print(row_index_age)
row_index_pclass = titanic_survival.loc[766, "Pclass"]
print(row_index_pclass)

OUT：

28.0
1

5、排序

new_titanic_survival = titanic_survival.sort_values("Age", ascending=False)
print(new_titanic_survival[:5])
print("----------")
titanic_reindexed = new_titanic_survival.reset_index(drop=True)
print(titanic_reindexed[:5])

OUT：

     PassengerId  Survived  Pclass                                  Name  \
630          631         1       1  Barkworth, Mr. Algernon Henry Wilson   
851          852         0       3                   Svensson, Mr. Johan   
493          494         0       1               Artagaveytia, Mr. Ramon   
96            97         0       1             Goldschmidt, Mr. George B   
116          117         0       3                  Connors, Mr. Patrick   

      Sex   Age  SibSp  Parch    Ticket     Fare Cabin Embarked  
630  male  80.0      0      0     27042  30.0000   A23        S  
851  male  74.0      0      0    347060   7.7750   NaN        S  
493  male  71.0      0      0  PC 17609  49.5042   NaN        C  
96   male  71.0      0      0  PC 17754  34.6542    A5        C  
116  male  70.5      0      0    370369   7.7500   NaN        Q  
----------
   PassengerId  Survived  Pclass                                  Name   Sex  \
0          631         1       1  Barkworth, Mr. Algernon Henry Wilson  male   
1          852         0       3                   Svensson, Mr. Johan  male   
2          494         0       1               Artagaveytia, Mr. Ramon  male   
3           97         0       1             Goldschmidt, Mr. George B  male   
4          117         0       3                  Connors, Mr. Patrick  male   

    Age  SibSp  Parch    Ticket     Fare Cabin Embarked  
0  80.0      0      0     27042  30.0000   A23        S  
1  74.0      0      0    347060   7.7750   NaN        S  
2  71.0      0      0  PC 17609  49.5042   NaN        C  
3  71.0      0      0  PC 17754  34.6542    A5        C  
4  70.5      0      0    370369   7.7500   NaN        Q  

6、自定义函数apply()
①查看第100个样本的数据

def hundredth_row(column):
    hundredth_item = column.loc[99]
    return hundredth_item

hundredth_row = titanic_survival.apply(hundredth_row)
print(hundredth_row)

OUT:

PassengerId                  100
Survived                       0
Pclass                         2
Name           Kantor, Mr. Sinai
Sex                         male
Age                           34
SibSp                          1
Parch                          0
Ticket                    244367
Fare                          26
Cabin                        NaN
Embarked                       S
dtype: object

②统计每一列的缺失值情况

def not_null_count(column):
    column_null = pd.isnull(column)
    null = column[column_null]
    return len(null)

column_null_count = titanic_survival.apply(not_null_count)
print(column_null_count)

OUT:

PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

③替换船舱等级称呼

def which_class(row):
    pclass = row['Pclass']
    if pd.isnull(pclass):
        return "Unknow"
    elif pclass == 1:
        return "First Class"
    elif pclass == 2:
        return "Second Class"
    elif pclass == 3:
        return "Third Class"

classes = titanic_survival.apply(which_class, axis=1)
print(classes[:10])

OUT:

0     Third Class
1     First Class
2     Third Class
3     First Class
4     Third Class
5     Third Class
6     First Class
7     Third Class
8     Third Class
9    Second Class
dtype: object

7、综合：查看获救几率与年龄之间的关系

#年龄分段
def generate_age_label(row):
    age = row["Age"]
    if pd.isnull(age):
        return "Unknow"
    elif age < 18:
        return "Minor"
    else:
        return "Adult"

age_label = titanic_survival.apply(generate_age_label, axis=1)
#分析获救几率与年龄之间的关系
titanic_survival['age_label'] = age_label
age_group_survival = titanic_survival.pivot_table(index="age_label", values="Survived")
print(age_group_survival)

OUT：

age_label  Survived        
Adult      0.381032
Minor      0.539823
Unknow     0.293785

---

附上：
数据分析处理库Pandas-数据读取
数据分析处理库Pandas-数据预处理
数据分析处理库Pandas-Series结构