32探索性数据分析-足球赛事数据集(含数据）

唐宇迪《python数据分析与机器学习实战》学习笔记
32探索性数据分析-足球赛事数据集
原始数据：链接，提取码：yypl

一、数据介绍、导入及查看

数据包含球员和裁判的信息，2012-2013年的比赛数据，总共设计球员2053名，裁判3147名，特征列表如下：

1.1 数据及模块导入

#from __future__ import absolute_import, division, print_function
import matplotlib as mpl
from matplotlib import pyplot as plt
from matplotlib.pyplot import GridSpec
import seaborn as sns
import numpy as np
import pandas as pd
import os, sys
from tqdm import tqdm
import warnings
warnings.filterwarnings('ignore')
sns.set_context("poster", font_scale=1.3)

#import missingno as msno
#import pandas_profiling

from sklearn.datasets import make_blobs
import time
df = pd.read_csv("redcard.csv.gz", compression='gzip')
print(df.shape)
df.head()

(146028, 28)

1.2 简单的统计：(count统计非空值个数)

df.describe().T

1.3查看数据类型：
(机器学习建模时只认识‘float’和‘int’型，其他类型需要映射转换一下，这里做探索分析就不用了)

df.dtypes

playerShort object
player object
club object
leagueCountry object
birthday object
height float64
weight float64
position object
games int64
victories int64
ties int64
defeats int64
goals int64
yellowCards int64
yellowReds int64
redCards int64
photoID object
rater1 float64
rater2 float64
refNum int64
refCountry int64
Alpha_3 object
meanIAT float64
nIAT float64
seIAT float64
meanExp float64
nExp float64
seExp float64
dtype: object

1.4 查看并提取列名

all_columns = df.columns.tolist()
all_columns

[‘playerShort’,
‘player’,
‘club’,
‘leagueCountry’,
‘birthday’,
‘height’,
‘weight’,
‘position’,
‘games’,
‘victories’,
‘ties’,
‘defeats’,
‘goals’,
‘yellowCards’,
‘yellowReds’,
‘redCards’,
‘photoID’,
‘rater1’,
‘rater2’,
‘refNum’,
‘refCountry’,
‘Alpha_3’,
‘meanIAT’,
‘nIAT’,
‘seIAT’,
‘meanExp’,
‘nExp’,
‘seExp’]

思考问题，加入一个运动员出现多次，计算时相当于其权重加强，所以可以用groupby解决这个问题：

print(df['height'].mean())
print(np.mean(df.groupby('playerShort').height.mean()))

181.93593798236887
181.74372848007872

二、数据切分模块(Tidy Data)

数据通常具有多特征高纬度，分析时统计指标不同，因此可以将其分为几个小的数据集单项分析。例如：单看球员、裁判，看球员-裁判关系，单看国家…

2.1切分球员数据

2.1.1数据切分

#切分出球员数据，及其相关特征数据
player_index = 'playerShort'  #球员ID
player_cols = [#'player', #球员名字丢弃，因为有对应ID了
               'birthday',
               'height',
               'weight',
               'position',
               'photoID',
               'rater1',
               'rater2',
              ]

2.1.2检测及去重

#数据检测，避免复制错误、重名等影响
all_cols_unique_players = df.groupby('playerShort').agg({
   col:'nunique' for col in player_cols})
all_cols_unique_players.head() #为1就是出现一次，为2就是重复了，这里数据干净       

all_cols_unique_players[all_cols_unique_players > 1].dropna().head()#去重

这里直接写了一个检测去重函数，主要是看key值重复没

def get_subgroup(dataframe, g_index, g_columns):
    
    """Helper function that creates a sub-table from the columns and runs a quick uniqueness test."""
    g = dataframe.groupby(g_index).agg({
   col:'nunique' for col in g_columns})
    if g[g > 1].dropna().shape[0] != 0:
        print("Warning: you probably assumed this had all unique values but it doesn't.")
    return dataframe.groupby(g_index).agg({
   col:'max' for col in g_columns})

函数调用

players = get_subgroup(df, player_index, player_cols)
players.head()

数据干净后就储存，这里增加储存函数：

def save_subgroup(dataframe, g_index, subgroup_name, prefix='raw_'):
    save_subgroup_filename = "".join([prefix, subgroup_name, ".csv.gz"])
    dataframe.to_csv(save_subgroup_filename, compression='gzip', encoding='UTF-8')
    test_df = pd.read_csv(save_subgroup_filename, compression='gzip', index_col=g_index, encoding='UTF-8')
    # Test that we recover what we send in
    if dataframe.equals(test_df):
        print("Test-passed: we recover the equivalent subgroup dataframe.")
    else:
        print("Warning -- equivalence test!!! Double-check.")

save_subgroup(players, player_index, "players")

Test-passed: we recover the equivalent subgroup dataframe. 储存成功

根据上面一套操作思路还可以切割其他数据：

2.2 切分俱乐部-国家关系

club_index = 'club'
club_cols = ['leagueCountry']
clubs = get_subgroup(df, club_index, club_cols)#检测去重
print(clubs.head())
print('-------------')
print(clubs['leagueCountry'].value_counts())#查看所属国家
save_subgroup(clubs, club_index, "clubs")#保存数据

club leagueCountry

1. FC Nürnberg Germany
2. FSV Mainz 05 Germany
   1899 Hoffenheim Germany
   AC Ajaccio France
   AFC Bournemouth England

---

England 48
Spain 27
France 22
Germany 21
Name: leagueCountry, dtype: int64
Test-passed: we recover the equivalent subgroup dataframe.

2.3 切分裁判-国家关系

referee_index = 'refNum'
referee_cols = ['refCountry'