python与数据挖掘上机实验pandas_python-pandas的一些小练习——东北大学大数据班数据挖掘实训Python基础二（2）...

因为本博客是直接导入markdown文件(jupter写的)生成的，所以会导致分不清代码和输出，如果大家需要清楚的代码和输出，就请下载以下html文件

pandasTrain.html

实训练习：练习使用pandas 包中的dataframe和方法。

1将数据读取为dataframe类型，命名为df

import pandas as pd

import numpy as np

df=pd.read_csv("C:\\Users\\zzh\\Desktop\\dataMiningExperment\\exp2\data\\flight.csv")

df.head()

date

hour

minute

dep

arr

dep_delay

arr_delay

carrier

flight

dest

plane

cancelled

time

dist

0

2011/12/1

14.0

31.0

1431.0

1730.0

-4.0

-9.0

XE

4319

SAV

N11121

0

104.0

851

1

2011/12/1

21.0

15.0

2115.0

2229.0

-10.0

-7.0

XE

4342

LRD

N11107

0

50.0

301

2

2011/12/1

10.0

47.0

1047.0

1429.0

20.0

17.0

XE

4434

RIC

N11113

0

146.0

1157

3

2011/12/1

19.0

24.0

1924.0

2234.0

-1.0

1.0

XE

4525

SAV

N11106

0

105.0

851

4

2011/12/1

7.0

18.0

718.0

1105.0

-2.0

-8.0

XE

4602

DTW

N11121

0

139.0

1075

2查看数据的形状[了解数据有多少行、多少列]

df.shape

(174, 14)

3查看数据的最开始的7条数据和尾部的3条数据

df.head(7)

date

hour

minute

dep

arr

dep_delay

arr_delay

carrier

flight

dest

plane

cancelled

time

dist

0

2011/12/1

14.0

31.0

1431.0

1730.0

-4.0

-9.0

XE

4319

SAV

N11121

0

104.0

851

1

2011/12/1

21.0

15.0

2115.0

2229.0

-10.0

-7.0

XE

4342

LRD

N11107

0

50.0

301

2

2011/12/1

10.0

47.0

1047.0

1429.0

20.0

17.0

XE

4434

RIC

N11113

0

146.0

1157

3

2011/12/1

19.0

24.0

1924.0

2234.0

-1.0

1.0

XE

4525

SAV

N11106

0

105.0

851

4

2011/12/1

7.0

18.0

718.0

1105.0

-2.0

-8.0

XE

4602

DTW

N11121

0

139.0

1075

5

2011/12/1

20.0

18.0

2018.0

2200.0

-2.0

-9.0

XE

4674

BNA

N11121

0

82.0

657

6

2011/12/2

21.0

20.0

2120.0

2230.0

-5.0

-6.0

XE

4342

LRD

N11113

0

53.0

301

df.tail(3)

date

hour

minute

dep

arr

dep_delay

arr_delay

carrier

flight

dest

plane

cancelled

time

dist

171

2011/12/30

11.0

14.0

1114.0

1141.0

9.0

6.0

XE

4323

HOB

N11107

0

73.0

501

172

2011/12/30

18.0

31.0

1831.0

2145.0

16.0

9.0

XE

4634

CLT

N11121

0

116.0

912

173

2011/12/31

14.0

8.0

1408.0

1705.0

2.0

-1.0

XE

4433

TYS

N11107

0

98.0

771

4查看数据的各字段的类型

df.dtypes

date object

hour float64

minute float64

dep float64

arr float64

dep_delay float64

arr_delay float64

carrier object

flight int64

dest object

plane object

cancelled int64

time float64

dist int64

dtype: object

5查看数据是否有缺失值

df.isna().any() #列级别的判断，只要该列有为空或者NA的元素，就为True，否则False isna()和isnull()一样

date False

hour True

minute True

dep True

arr True

dep_delay True

arr_delay True

carrier False

flight False

dest False

plane False

cancelled False

time True

dist False

dtype: bool

6如果有缺失值，进行补0操作

df=df.fillna(0)

df.isna().any()

date False

hour False

minute False

dep False

arr False

dep_delay False

arr_delay False

carrier False

flight False

dest False

plane False

cancelled False

time False

dist False

dtype: bool

7从df中按字段取出‘ date’’dist’和‘ flight’形成新的df1

df1 = df.loc[:,['date','dist','flight']]

df1.head()

date

dist

flight

0

2011/12/1

851

4319

1

2011/12/1

301

4342

2

2011/12/1

1157

4434

3

2011/12/1

851

4525

4

2011/12/1

1075

4602

8从df中按位置选取行为3和4，列为0,1的数据形成新的df2

df2 = df.iloc[[3,4],[0,1]]

df2

date

hour

3

2011/12/1

19.0

4

2011/12/1

7.0

9在df中添加一列‘ low_dest’，内容是‘ dest’这列的字符串的小写形式

df['low_dest'] = df['dest'].str.lower()

df.head()

date

hour

minute

dep

arr

dep_delay

arr_delay

carrier

flight

dest

plane

cancelled

time

dist

low_dest

0

2011/12/1

14.0

31.0

1431.0

1730.0

-4.0

-9.0

XE

4319

SAV

N11121

0

104.0

851

sav

1

2011/12/1

21.0

15.0

2115.0

2229.0

-10.0

-7.0

XE

4342

LRD

N11107

0

50.0

301

lrd

2

2011/12/1

10.0

47.0

1047.0

1429.0

20.0

17.0

XE

4434

RIC

N11113

0

146.0

1157

ric

3

2011/12/1

19.0

24.0

1924.0

2234.0

-1.0

1.0

XE

4525

SAV

N11106

0

105.0

851

sav

4

2011/12/1

7.0

18.0

718.0

1105.0

-2.0

-8.0

XE

4602

DTW

N11121

0

139.0

1075

dtw

10从df中选取‘ dist’大于1000并且小于1200的数据作为一个新的df3

df3 = df[(df.dist>1000) & (df.dist<1200)]

df3.head()

date

hour

minute

dep

arr

dep_delay

arr_delay

carrier

flight

dest

plane

cancelled

time

dist

low_dest

2

2011/12/1

10.0

47.0

1047.0

1429.0

20.0

17.0

XE

4434

RIC

N11113

0

146.0

1157

ric

4

2011/12/1

7.0

18.0

718.0

1105.0

-2.0

-8.0

XE

4602

DTW

N11121

0

139.0

1075

dtw

10

2011/12/2

7.0

13.0

713.0

1058.0

-2.0

-6.0

XE

4501

PIT

N11107

0

146.0

1117

pit

14

2011/12/3

19.0

1.0

1901.0

2230.0

-1.0

-14.0

XE

4509

RIC

N11113

0

134.0

1157

ric

16

2011/12/4

16.0

8.0

1608.0

2004.0

18.0

40.0

XE

4477

RDU

N11121

0

123.0

1042

rdu

11从df中选取‘ time’大于100的数据做为新的df4

df4 = df[df.time>100]

df4.head()

date

hour

minute

dep

arr

dep_delay

arr_delay

carrier

flight

dest

plane

cancelled

time

dist

low_dest

0

2011/12/1

14.0

31.0

1431.0

1730.0

-4.0

-9.0

XE

4319

SAV

N11121

0

104.0

851

sav

2

2011/12/1

10.0

47.0

1047.0

1429.0

20.0

17.0

XE

4434

RIC

N11113

0

146.0

1157

ric

3

2011/12/1

19.0

24.0

1924.0

2234.0

-1.0

1.0

XE

4525

SAV

N11106

0

105.0

851

sav

4

2011/12/1

7.0

18.0

718.0

1105.0

-2.0

-8.0

XE

4602

DTW

N11121

0

139.0

1075

dtw

7

2011/12/2

10.0

25.0

1025.0

1352.0

-5.0

-2.0

XE

4366

CLT

N11121

0

120.0

912

clt

12将df3和df4合并为一个dataframe，命名为df5

df5 = pd.concat([df3,df4],axis = 1) #横向合并 axis决定合并方向 ,axis = 1横向合并 axis = 0竖向合并

df5.head()

date

hour

minute

dep

arr

dep_delay

arr_delay

carrier

flight

dest

...

dep_delay

arr_delay

carrier

flight

dest

plane

cancelled

time

dist

low_dest

0

NaN

NaN

NaN

NaN

NaN

NaN

NaN

NaN

NaN

NaN

...

-4.0

-9.0

XE

4319.0

SAV

N11121

0.0

104.0

851.0

sav

2

2011/12/1

10.0

47.0

1047.0

1429.0

20.0

17.0

XE

4434.0

RIC

...

20.0

17.0

XE

4434.0

RIC

N11113

0.0

146.0

1157.0

ric

3

NaN

NaN

NaN

NaN

NaN

NaN

NaN

NaN

NaN

NaN

...

-1.0

1.0

XE

4525.0

SAV

N11106

0.0

105.0

851.0

sav

4

2011/12/1

7.0

18.0

718.0

1105.0

-2.0

-8.0

XE

4602.0

DTW

...

-2.0

-8.0

XE

4602.0

DTW

N11121

0.0

139.0

1075.0

dtw

7

NaN

NaN

NaN

NaN

NaN

NaN

NaN

NaN

NaN

NaN

...

-5.0

-2.0

XE

4366.0

CLT

N11121

0.0

120.0

912.0

clt

5 rows × 30 columns

df3.shape

(43, 15)

df4.shape

(85, 15)

df5.shape #15+15=30

(86, 30)

pd.concat([df3,df4],axis = 0).shape #竖向合并 43+85=128

(128, 15)

13取出df3中的‘ time’,‘dist’ 这两列做为df6， 求出df6中每列中最大值与最小值的差值(运用lambda函数的方式进行求解 )

df6 = df3.loc[:,['time','dist']]

df6.head()

df6.apply(lambda x : x.max() - x.min())

14将df根据“ dest”分组，统计各目的地，都有多少条数据，命名为“ dest_count”，然后将“ dest”和“ dest_count”组成新的dataframe命名为df7

df7 = df.groupby('dest',as_index = False)['dest'].agg({'dest_count':np.size})

df7.head()

15根据dep_delay不为0，统计df中有多少次航班起飞延误

df[df.dep_delay!=0].shape[0]

16利用plane字段，统计df中不同航班的飞行次数，以plane和plane_cout格式保存为df8

df8 = df.groupby('plane',as_index = False)['plane'].agg({'dest_count':np.size})

df8.head()

17利用carrier字段，计算df中carrier字段不同属性数据的数据量的比值(如：XE的数据量与MQ的数据量的比值)

df9=df.groupby(['carrier'],as_index=False)['carrier'].size().reset_index(name='carrier_count')

df9.head()

print("XE的数据量与MQ的数据量的比值: ", df9.iat[1,1]/df9.iat[0,1])

18分别将df7和 df8保存到本地文件，文件名分别为df7.csv和df8.csv

df7.to_csv("C:\\Users\\zzh\\Desktop\\dataMiningExperment\\exp2\\df7.csv")

df8.to_csv("C:\\Users\\zzh\\Desktop\\dataMiningExperment\\exp2\\df78.csv")