复习:在前面我们已经学习了Pandas基础,第二章我们开始进入数据分析的业务部分,在第二章第一节的内容中,我们学习了 数据的清洗 ,这一部分十分重要,只有数据变得相对干净,我们之后对数据的分析才可以更有力。而这一节,我们要做的是数据重构,数据重构依旧属于数据理解(准备)的范围。
开始之前,导入numpy、pandas包和数据
import pandas as pd
import numpy as np
df= pd. read_csv( 'F:\Datewheel资料\hands-on-data-analysis-master\hands-on-data-analysis-master\第二章项目集合\data/train-left-up.csv' )
df. head( )
PassengerId Survived Pclass Name 0 1 0 3 Braund, Mr. Owen Harris 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... 2 3 1 3 Heikkinen, Miss. Laina 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) 4 5 0 3 Allen, Mr. William Henry
2 第二章:数据重构
第一部分:数据规整
2.4 数据的合并
2.4.1 任务一:将data文件夹里面的所有数据都载入,观察数据的之间的关系
df1= pd. read_csv( 'F:\Datewheel资料\hands-on-data-analysis-master\hands-on-data-analysis-master\第二章项目集合\data/train-left-down.csv' )
df1. head( )
PassengerId Survived Pclass Name 0 440 0 2 Kvillner, Mr. Johan Henrik Johannesson 1 441 1 2 Hart, Mrs. Benjamin (Esther Ada Bloomfield) 2 442 0 3 Hampe, Mr. Leon 3 443 0 3 Petterson, Mr. Johan Emil 4 444 1 2 Reynaldo, Ms. Encarnacion
df2= pd. read_csv( 'F:\Datewheel资料\hands-on-data-analysis-master\hands-on-data-analysis-master\第二章项目集合\data/train-right-up.csv' )
df2. head( )
Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 male 22.0 1.0 0.0 A/5 21171 7.2500 NaN S 1 female 38.0 1.0 0.0 PC 17599 71.2833 C85 C 2 female 26.0 0.0 0.0 STON/O2. 3101282 7.9250 NaN S 3 female 35.0 1.0 0.0 113803 53.1000 C123 S 4 male 35.0 0.0 0.0 373450 8.0500 NaN S
df3= pd. read_csv( 'F:\Datewheel资料\hands-on-data-analysis-master\hands-on-data-analysis-master\第二章项目集合\data/train-right-down.csv' )
df3. head( )
Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 male 31.0 0 0 C.A. 18723 10.500 NaN S 1 female 45.0 1 1 F.C.C. 13529 26.250 NaN S 2 male 20.0 0 0 345769 9.500 NaN S 3 male 25.0 1 0 347076 7.775 NaN S 4 female 28.0 0 0 230434 13.000 NaN S
【提示】结合之前我们加载的train.csv数据,大致预测一下上面的数据是什么
将原有的train. csv拆分成了四个部分
2.4.2:任务二:使用concat方法:将数据train-left-up.csv和train-right-up.csv横向合并为一张表,并保存这张表为result_up
result_up= pd. concat( [ df, df2] , axis= 1 )
result_up. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 1.0 0.0 3.0 Braund, Mr. Owen Harris male 22.0 1.0 0.0 A/5 21171 7.2500 NaN S 1 2.0 1.0 1.0 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1.0 0.0 PC 17599 71.2833 C85 C 2 3.0 1.0 3.0 Heikkinen, Miss. Laina female 26.0 0.0 0.0 STON/O2. 3101282 7.9250 NaN S 3 4.0 1.0 1.0 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1.0 0.0 113803 53.1000 C123 S 4 5.0 0.0 3.0 Allen, Mr. William Henry male 35.0 0.0 0.0 373450 8.0500 NaN S
2.4.3 任务三:使用concat方法:将train-left-down和train-right-down横向合并为一张表,并保存这张表为result_down。然后将上边的result_up和result_down纵向合并为result。
result_down= pd. concat( [ df1, df3] , axis= 1 )
result_down. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 440 0 2 Kvillner, Mr. Johan Henrik Johannesson male 31.0 0 0 C.A. 18723 10.500 NaN S 1 441 1 2 Hart, Mrs. Benjamin (Esther Ada Bloomfield) female 45.0 1 1 F.C.C. 13529 26.250 NaN S 2 442 0 3 Hampe, Mr. Leon male 20.0 0 0 345769 9.500 NaN S 3 443 0 3 Petterson, Mr. Johan Emil male 25.0 1 0 347076 7.775 NaN S 4 444 1 2 Reynaldo, Ms. Encarnacion female 28.0 0 0 230434 13.000 NaN S
result= pd. concat( [ result_up, result_down] )
result. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 1.0 0.0 3.0 Braund, Mr. Owen Harris male 22.0 1.0 0.0 A/5 21171 7.2500 NaN S 1 2.0 1.0 1.0 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1.0 0.0 PC 17599 71.2833 C85 C 2 3.0 1.0 3.0 Heikkinen, Miss. Laina female 26.0 0.0 0.0 STON/O2. 3101282 7.9250 NaN S 3 4.0 1.0 1.0 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1.0 0.0 113803 53.1000 C123 S 4 5.0 0.0 3.0 Allen, Mr. William Henry male 35.0 0.0 0.0 373450 8.0500 NaN S
2.4.4 任务四:使用DataFrame自带的方法join方法和append:完成任务二和任务三的任务
result_up= df. join( df2)
result_up. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 1 0 3 Braund, Mr. Owen Harris male 22.0 1.0 0.0 A/5 21171 7.2500 NaN S 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1.0 0.0 PC 17599 71.2833 C85 C 2 3 1 3 Heikkinen, Miss. Laina female 26.0 0.0 0.0 STON/O2. 3101282 7.9250 NaN S 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1.0 0.0 113803 53.1000 C123 S 4 5 0 3 Allen, Mr. William Henry male 35.0 0.0 0.0 373450 8.0500 NaN S
result_down= df1. join( df3)
result_down. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 440 0 2 Kvillner, Mr. Johan Henrik Johannesson male 31.0 0 0 C.A. 18723 10.500 NaN S 1 441 1 2 Hart, Mrs. Benjamin (Esther Ada Bloomfield) female 45.0 1 1 F.C.C. 13529 26.250 NaN S 2 442 0 3 Hampe, Mr. Leon male 20.0 0 0 345769 9.500 NaN S 3 443 0 3 Petterson, Mr. Johan Emil male 25.0 1 0 347076 7.775 NaN S 4 444 1 2 Reynaldo, Ms. Encarnacion female 28.0 0 0 230434 13.000 NaN S
result= result_up. append( result_down)
result. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 1 0 3 Braund, Mr. Owen Harris male 22.0 1.0 0.0 A/5 21171 7.2500 NaN S 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1.0 0.0 PC 17599 71.2833 C85 C 2 3 1 3 Heikkinen, Miss. Laina female 26.0 0.0 0.0 STON/O2. 3101282 7.9250 NaN S 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1.0 0.0 113803 53.1000 C123 S 4 5 0 3 Allen, Mr. William Henry male 35.0 0.0 0.0 373450 8.0500 NaN S
2.4.5 任务五:使用Panads的merge方法和DataFrame的append方法:完成任务二和任务三的任务
result_up= pd. merge( df, df2, left_index= True , right_index= True )
result_up. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 1 0 3 Braund, Mr. Owen Harris male 22.0 1.0 0.0 A/5 21171 7.2500 NaN S 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1.0 0.0 PC 17599 71.2833 C85 C 2 3 1 3 Heikkinen, Miss. Laina female 26.0 0.0 0.0 STON/O2. 3101282 7.9250 NaN S 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1.0 0.0 113803 53.1000 C123 S 4 5 0 3 Allen, Mr. William Henry male 35.0 0.0 0.0 373450 8.0500 NaN S
result_down= pd. merge( df1, df3, left_index= True , right_index= True )
result_down. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 440 0 2 Kvillner, Mr. Johan Henrik Johannesson male 31.0 0 0 C.A. 18723 10.500 NaN S 1 441 1 2 Hart, Mrs. Benjamin (Esther Ada Bloomfield) female 45.0 1 1 F.C.C. 13529 26.250 NaN S 2 442 0 3 Hampe, Mr. Leon male 20.0 0 0 345769 9.500 NaN S 3 443 0 3 Petterson, Mr. Johan Emil male 25.0 1 0 347076 7.775 NaN S 4 444 1 2 Reynaldo, Ms. Encarnacion female 28.0 0 0 230434 13.000 NaN S
result= result_up. append( result_down)
result. head( )
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 1 0 3 Braund, Mr. Owen Harris male 22.0 1.0 0.0 A/5 21171 7.2500 NaN S 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1.0 0.0 PC 17599 71.2833 C85 C 2 3 1 3 Heikkinen, Miss. Laina female 26.0 0.0 0.0 STON/O2. 3101282 7.9250 NaN S 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1.0 0.0 113803 53.1000 C123 S 4 5 0 3 Allen, Mr. William Henry male 35.0 0.0 0.0 373450 8.0500 NaN S
【思考】对比merge、join以及concat的方法的不同以及相同。思考一下在任务四和任务五的情况下,为什么都要求使用DataFrame的append方法,如何只要求使用merge或者join可不可以完成任务四和任务五呢?
merge和join只能是横向拼接,所以单独使用merge或join方法不能实现纵向拼接 merge、join、concat都能实现横向拼接,但concat还能通过指定axis的取值实现纵向连接,其他两种方法不行 mergehejoin方法可以用on参数来指定来指定某一个对象作为key实现连接
2.4.6 任务六:完成的数据保存为result.csv
result. to_csv( 'F:\Datewheel资料\hands-on-data-analysis-master\hands-on-data-analysis-master\第二章项目集合\data/result.csv' )
2.5 换一种角度看数据
2.5.1 任务一:将我们的数据变为Series类型的数据
stack: 将数据从”表格结构“变成”花括号结构“,即将其列索引变成行索引。 unstack: 数据从”花括号结构“变成”表格结构“,即要将其中一层的行索引变成列索引。如果是多层索引,则以上函数是针对内层索引(这里是store)。利用level可以选择具体哪层索引。
result. stack( ) . head( )
0 PassengerId 1
Survived 0
Pclass 3
Name Braund, Mr. Owen Harris
Sex male
dtype: object
第二节
import pandas as pd
import numpy as np
result= pd. read_csv( 'F:\Datewheel资料\hands-on-data-analysis-master\hands-on-data-analysis-master\第二章项目集合\data/result.csv' )
result. head( )
Unnamed: 0 PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked 0 0 1 0 3 Braund, Mr. Owen Harris male 22.0 1.0 0.0 A/5 21171 7.2500 NaN S 1 1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1.0 0.0 PC 17599 71.2833 C85 C 2 2 3 1 3 Heikkinen, Miss. Laina female 26.0 0.0 0.0 STON/O2. 3101282 7.9250 NaN S 3 3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1.0 0.0 113803 53.1000 C123 S 4 4 5 0 3 Allen, Mr. William Henry male 35.0 0.0 0.0 373450 8.0500 NaN S
2.4.2:任务二:计算泰坦尼克号男性与女性的平均票价
df= result[ 'Fare' ] . groupby( result[ 'Sex' ] )
df. head( )
fs= df. mean( )
2.4.3:任务三:统计泰坦尼克号中男女的存活人数
df1= result[ 'Survived' ] . groupby( result[ 'Sex' ] )
ss= df1. sum ( )
2.4.4:任务四:计算客舱不同等级的存活人数
df2= result[ 'Survived' ] . groupby( result[ 'Pclass' ] )
df2. sum ( )
Pclass
1 136
2 87
3 119
Name: Survived, dtype: int64
从男女平均票价以及存活人数来看,可以发现平均票价比较高的女性存货数量也比较多,可以猜测票价的高低会影响存活。
从不同等级客舱的存活人数来看,等级最高的存活人数也最多。
2.4.5:任务五:统计在不同等级的票中的不同年龄的船票花费的平均值
result. groupby( [ 'Pclass' , 'Age' ] ) [ 'Fare' ] . mean( )
Pclass Age
1 0.92 151.5500
2.00 151.5500
4.00 81.8583
11.00 120.0000
14.00 120.0000
...
3 61.00 6.2375
63.00 9.5875
65.00 7.7500
70.50 7.7500
74.00 7.7750
Name: Fare, Length: 182, dtype: float64
2.4.6:任务六:将任务二和任务三的数据合并,并保存到sex_fare_survived.csv
sex_fare_survived= pd. merge( fs, ss, on= 'Sex' )
sex_fare_survived. head( )
sex_fare_survived. to_csv( 'F:\Datewheel资料\hands-on-data-analysis-master\hands-on-data-analysis-master\第二章项目集合\data/sex_fare_survived.csv' )
2.4.7:任务七:得出不同年龄的总的存活人数,然后找出存活人数的最高的年龄,最后计算存活人数最高的存活率(存活人数/总人数)
sa= result. groupby( [ 'Age' ] ) [ 'Survived' ] . sum ( )
sa. head( )
Age
0.42 1
0.67 1
0.75 2
0.83 2
0.92 1
Name: Survived, dtype: int64
sa[ sa. values== sa. max ( ) ]
Age
24.0 15
Name: Survived, dtype: int64
result[ 'Survived' ] . sum ( )
342
result. info( )
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 13 columns):
Unnamed: 0 891 non-null int64
PassengerId 891 non-null int64
Survived 891 non-null int64
Pclass 891 non-null int64
Name 891 non-null object
Sex 891 non-null object
Age 714 non-null float64
SibSp 891 non-null float64
Parch 891 non-null float64
Ticket 891 non-null object
Fare 891 non-null float64
Cabin 204 non-null object
Embarked 889 non-null object
dtypes: float64(4), int64(4), object(5)
memory usage: 90.6+ KB
spercent= sa[ sa. values== sa. max ( ) ] / result[ 'Survived' ] . sum ( )
print ( '存货人数最高的存活率为:' , str ( spercent) )
存货人数最高的存活率为: Age
24.0 0.04386
Name: Survived, dtype: float64