复习:在前面我们已经学习了Pandas基础,第二章我们开始进入数据分析的业务部分,在第二章第一节的内容中,我们学习了 数据的清洗 ,这一部分十分重要,只有数据变得相对干净,我们之后对数据的分析才可以更有力。而这一节,我们要做的是数据重构,数据重构依旧属于数据理解(准备)的范围。
开始之前,导入numpy、pandas包和数据
import numpy as np
import pandas as pd
df= pd. read_csv( "data/train-left-up.csv" )
df. head( )
PassengerId Survived Pclass Name 0 1 0 3 Braund, Mr. Owen Harris 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... 2 3 1 3 Heikkinen, Miss. Laina 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) 4 5 0 3 Allen, Mr. William Henry
2 第二章:数据重构
2.4 数据的合并
2.4.1 任务一:将data文件夹里面的所有数据都载入,观察数据的之间的关系
text_left_up = pd. read_csv( "data/train-left-up.csv" )
text_left_down = pd. read_csv( "data/train-left-down.csv" )
text_right_up = pd. read_csv( "data/train-right-up.csv" )
text_right_down = pd. read_csv( "data/train-right-down.csv" )
text_left_up. head( )
PassengerId Survived Pclass Name 0 1 0 3 Braund, Mr. Owen Harris 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... 2 3 1 3 Heikkinen, Miss. Laina 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) 4 5 0 3 Allen, Mr. William Henry
text_left_down. head( )
PassengerId Survived Pclass Name 0 440 0 2 Kvillner, Mr. Johan Henrik Johannesson 1 441 1 2 Hart, Mrs. Benjamin (Esther Ada Bloomfield) 2 442 0 3 Hampe, Mr. Leon 3 443 0 3 Petterson, Mr. Johan Emil 4 444 1 2 Reynaldo, Ms. Encarnacion
text_right_down. head( )
Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 male 31.0 0 0 C.A. 18723 10.500 NaN S 1 female 45.0 1 1 F.C.C. 13529 26.250 NaN S 2 male 20.0 0 0 345769 9.500 NaN S 3 male 25.0 1 0 347076 7.775 NaN S 4 female 28.0 0 0 230434 13.000 NaN S
text_right_up. head( )
Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 male 22.0 1 0 A/5 21171 7.2500 NaN S 1 female 38.0 1 0 PC 17599 71.2833 C85 C 2 female 26.0 0 0 STON/O2. 3101282 7.9250 NaN S 3 female 35.0 1 0 113803 53.1000 C123 S 4 male 35.0 0 0 373450 8.0500 NaN S
【提示】结合之前我们加载的train.csv数据,大致预测一下上面的数据是什么
2.4.2:任务二:使用concat方法:将数据train-left-up.csv和train-right-up.csv横向合并为一张表,并保存这张表为result_up
result_up= pd. concat( [ text_left_up, text_right_up] , axis= 1 )
result_up. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 1 0 3 Braund, Mr. Owen Harris male 22.0 1 0 A/5 21171 7.2500 NaN S 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 0 PC 17599 71.2833 C85 C 2 3 1 3 Heikkinen, Miss. Laina female 26.0 0 0 STON/O2. 3101282 7.9250 NaN S 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 0 113803 53.1000 C123 S 4 5 0 3 Allen, Mr. William Henry male 35.0 0 0 373450 8.0500 NaN S
2.4.3 任务三:使用concat方法:将train-left-down和train-right-down横向合并为一张表,并保存这张表为result_down。然后将上边的result_up和result_down纵向合并为result。
result_down = pd. concat( [ text_left_down, text_right_down] , axis= 1 )
result = pd. concat( [ result_up, result_down] )
result. shape
(891, 12)
2.4.4 任务四:使用DataFrame自带的方法join方法和append:完成任务二和任务三的任务
resul_up = text_left_up. join( text_right_up)
result_down = text_left_down. join( text_right_down)
result = result_up. append( result_down)
result. head( )
2.4.5 任务五:使用Panads的merge方法和DataFrame的append方法:完成任务二和任务三的任务
result_up = pd. merge( text_left_up, text_right_up, left_index= True , right_index= True )
result_down = pd. merge( text_left_down, text_right_down, left_index= True , right_index= True )
result = result_up. append( result_down)
result. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 1 0 3 Braund, Mr. Owen Harris male 22.0 1 0 A/5 21171 7.2500 NaN S 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 0 PC 17599 71.2833 C85 C 2 3 1 3 Heikkinen, Miss. Laina female 26.0 0 0 STON/O2. 3101282 7.9250 NaN S 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 0 113803 53.1000 C123 S 4 5 0 3 Allen, Mr. William Henry male 35.0 0 0 373450 8.0500 NaN S
【思考】对比merge、join以及concat的方法的不同以及相同。思考一下在任务四和任务五的情况下,为什么都要求使用DataFrame的append方法,如何只要求使用merge或者join可不可以完成任务四和任务五呢?
2.4.6 任务六:完成的数据保存为result.csv
result. to_csv( "result.csv" )
2.5 换一种角度看数据
2.5.1 任务一:将我们的数据变为Series类型的数据
text = pd. read_csv( 'result.csv' )
unit_result= text. stack( )
unit_result
0 Unnamed: 0 0
PassengerId 1
Survived 0
Pclass 3
Name Braund, Mr. Owen Harris
...
890 SibSp 0
Parch 0
Ticket 370376
Fare 7.75
Embarked Q
Length: 10717, dtype: object
unit_result. to_csv( 'unit_result.csv' )
test = pd. read_csv( 'unit_result.csv' )
test
Unnamed: 0 Unnamed: 1 0 0 0 Unnamed: 0 0 1 0 PassengerId 1 2 0 Survived 0 3 0 Pclass 3 4 0 Name Braund, Mr. Owen Harris ... ... ... ... 10712 890 SibSp 0 10713 890 Parch 0 10714 890 Ticket 370376 10715 890 Fare 7.75 10716 890 Embarked Q
10717 rows × 3 columns
复习:在前面我们已经学习了Pandas基础,第二章我们开始进入数据分析的业务部分,在第二章第一节的内容中,我们学习了 数据的清洗 ,这一部分十分重要,只有数据变得相对干净,我们之后对数据的分析才可以更有力。而这一节,我们要做的是数据重构,数据重构依旧属于数据理解(准备)的范围。
开始之前,导入numpy、pandas包和数据
import numpy as np
import pandas as pd
df = pd. read_csv( 'result.csv' )
df. head( )
Unnamed: 0 PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 0 1 0 3 Braund, Mr. Owen Harris male 22.0 1 0 A/5 21171 7.2500 NaN S 1 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 0 PC 17599 71.2833 C85 C 2 2 3 1 3 Heikkinen, Miss. Laina female 26.0 0 0 STON/O2. 3101282 7.9250 NaN S 3 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 0 113803 53.1000 C123 S 4 4 5 0 3 Allen, Mr. William Henry male 35.0 0 0 373450 8.0500 NaN S
2 第二章:数据重构
第一部分:数据聚合与运算
2.6 数据运用
2.6.1 任务一:通过教材《Python for Data Analysis》P303、Google or anything来学习了解GroupBy机制
2.6.2:任务二:计算泰坦尼克号男性与女性的平均票价
list ( df. groupby( 'Sex' ) ) [ 0 ]
('female',
Unnamed: 0 PassengerId Survived Pclass \
1 1 2 1 1
2 2 3 1 3
3 3 4 1 1
8 8 9 1 3
9 9 10 1 2
.. ... ... ... ...
880 441 881 1 2
882 443 883 0 3
885 446 886 0 3
887 448 888 1 1
888 449 889 0 3
Name Sex Age SibSp \
1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1
2 Heikkinen, Miss. Laina female 26.0 0
3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1
8 Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg) female 27.0 0
9 Nasser, Mrs. Nicholas (Adele Achem) female 14.0 1
.. ... ... ... ...
880 Shelley, Mrs. William (Imanita Parrish Hall) female 25.0 0
882 Dahlberg, Miss. Gerda Ulrika female 22.0 0
885 Rice, Mrs. William (Margaret Norton) female 39.0 0
887 Graham, Miss. Margaret Edith female 19.0 0
888 Johnston, Miss. Catherine Helen "Carrie" female NaN 1
Parch Ticket Fare Cabin Embarked
1 0 PC 17599 71.2833 C85 C
2 0 STON/O2. 3101282 7.9250 NaN S
3 0 113803 53.1000 C123 S
8 2 347742 11.1333 NaN S
9 0 237736 30.0708 NaN C
.. ... ... ... ... ...
880 1 230433 26.0000 NaN S
882 0 7552 10.5167 NaN S
885 5 382652 29.1250 NaN Q
887 0 112053 30.0000 B42 S
888 2 W./C. 6607 23.4500 NaN S
[314 rows x 13 columns])
df. groupby( 'Sex' ) [ 'Age' ] . describe( )
count mean std min 25% 50% 75% max Sex female 261.0 27.915709 14.110146 0.75 18.0 27.0 37.0 63.0 male 453.0 30.726645 14.678201 0.42 21.0 29.0 39.0 80.0
在了解GroupBy机制之后,运用这个机制完成一系列的操作,来达到我们的目的。
下面通过几个任务来熟悉GroupBy机制。
a= df. groupby( 'Sex' ) [ 'Fare' ] . mean( )
a
Sex
female 44.479818
male 25.523893
Name: Fare, dtype: float64
2.6.3:任务三:统计泰坦尼克号中男女的存活人数
b= df. groupby( 'Sex' ) [ 'Survived' ] . sum ( )
b
Sex
female 233
male 109
Name: Survived, dtype: int64
2.6.4:任务四:计算客舱不同等级的存活人数
df. groupby( 'Pclass' ) [ 'Survived' ] . sum ( )
Pclass
1 136
2 87
3 119
Name: Survived, dtype: int64
【提示: 】表中的存活那一栏,可以发现如果还活着记为1,死亡记为0
【思考 】从数据分析的角度,上面的统计结果可以得出那些结论
【思考】从任务二到任务三中,这些运算可以通过agg()函数来同时计算。并且可以使用rename函数修改列名。你可以按照提示写出这个过程吗?
text. groupby( 'Sex' ) . agg( { 'Fare' : 'mean' , 'Survived' : 'sum' } ) . rename( columns= { 'Fare' : 'mean_fare' , 'Survived' : 'sum_Survived' } )
mean_fare sum_Survived Sex female 44.479818 233 male 25.523893 109
2.6.5:任务五:统计在不同等级的票中的不同年龄的船票花费的平均值
text. groupby( [ 'Pclass' , 'Age' ] ) [ 'Fare' ] . mean( ) . head( )
Pclass Age
1 0.92 151.5500
2.00 151.5500
4.00 81.8583
11.00 120.0000
14.00 120.0000
Name: Fare, dtype: float64
2.6.6:任务六:将任务二和任务三的数据合并,并保存到sex_fare_survived.csv
result = pd. merge( a, b, on= 'Sex' )
result
Fare Survived Sex female 44.479818 233 male 25.523893 109
2.6.7:任务七:得出不同年龄的总的存活人数,然后找出存活人数最多的年龄段,最后计算存活人数最高的存活率(存活人数/总人数)
survived_age = df. groupby( [ 'Age' ] ) [ 'Survived' ] . sum ( )
survived_age. head( )
Age
0.42 1
0.67 1
0.75 2
0.83 2
0.92 1
Name: Survived, dtype: int64
max ( survived_age)
survived_age[ survived_age. values == max ( survived_age) ]
Age
24.0 15
Name: Survived, dtype: int64
rate= max ( survived_age) / sum ( df[ 'Survived' ] )
'最大存活率:{}' . format ( rate)
'最大存活率:0.043859649122807015'