2021-03-19

各种数据科学以及可视化库:

数据科学库 pandas、numpy、scipy；
可视化库 matplotlib、seabon；

#coding:utf-8
#导入warnings包，利用过滤器来实现忽略警告语句。
import warnings
warnings.filterwarnings('ignore')
import missingno as msno
import pandas as pd
from pandas import DataFrame
import matplotlib.pyplot as plt 
import seaborn as sns
import numpy as np

import os
import gc
import math

import pandas as pd
import numpy as np

载入数据：

载入训练集和测试集；
简略观察数据(head()+shape)；
导入训练集train.csv。

import pandas as pd
from pandas import DataFrame, Series
import matplotlib.pyplot as plt
Train_data = pd.read_csv('./train.csv')

train = pd.read_csv('/Users/apple/Desktop/Tianchi/train.csv')
test=pd.read_csv('/Users/apple/Desktop/Tianchi/testA.csv')
train.head()

输出结果：

id	heartbeat_signals	label
0	0	0.9912297987616655,0.9435330436439665,0.764677...	0.0
1	1	0.9714822034884503,0.9289687459588268,0.572932...	0.0
2	2	1.0,0.9591487564065292,0.7013782792997189,0.23...	2.0
3	3	0.9757952826275774,0.9340884687738161,0.659636...	0.0
4	4	0.0,0.055816398940721094,0.26129357194994196,0...	2.0

导入测试集 testA.csv。

import pandas as pd
from pandas import DataFrame, Series
import matplotlib.pyplot as plt 
Test_data = pd.read_csv('./testA.csv')

train = pd.read_csv('/Users/apple/Desktop/Tianchi/train.csv')
test=pd.read_csv('/Users/apple/Desktop/Tianchi/testA.csv')
train.head()

data.head().append(data.tail())——观察首尾数据

data.shape——观察数据集的行列信息

数据总览:

describe()来熟悉数据的相关统计量：describe种有每列的统计量，个数count、平均值mean、方差std、最小值min、中位数25% 50% 75% 、以及最大值
info()来熟悉数据类型：info 通过info来了解数据每列的type，有助于了解是否存在除了nan以外的特殊符号异常
data.describe()——获取数据的相关统计量
data.info()——获取数据类型

Train_data.describe()

id	label
count	100000.000000	100000.000000
mean	49999.500000	0.856960
std	28867.657797	1.217084
min	0.000000	0.000000
25%	24999.750000	0.000000
50%	49999.500000	0.000000
75%	74999.250000	2.000000
max	99999.000000	3.000000

Train_data.info()

<bound method DataFrame.info of           id                              heartbeat_signals  label
0          0  0.9912297987616655,0.9435330436439665,0.764677...    0.0
1          1  0.9714822034884503,0.9289687459588268,0.572932...    0.0
2          2  1.0,0.9591487564065292,0.7013782792997189,0.23...    2.0
3          3  0.9757952826275774,0.9340884687738161,0.659636...    0.0
4          4  0.0,0.055816398940721094,0.26129357194994196,0...    2.0
...      ...                                                ...    ...
99995  99995  1.0,0.677705342021188,0.22239242747868546,0.25...    0.0
99996  99996  0.9268571578157265,0.9063471198026871,0.636993...    2.0
99997  99997  0.9258351628306013,0.5873839035878395,0.633226...    3.0
99998  99998  1.0,0.9947621698382489,0.8297017704865509,0.45...    2.0
99999  99999  0.9259994004527861,0.916476635326053,0.4042900...    0.0

[100000 rows x 3 columns]>

判断数据缺失和异常

data.isnull().sum()——查看每列的存在nan情况

查看trian每列的存在nan情况
下面展示一些 内联代码片。

Train_data.isnull().sum()

result:

id                   0
heartbeat_signals    0
label                0
dtype: int64

查看testA每列的存在nan情况

Test_data.isnull().sum()

id                   0
heartbeat_signals    0
dtype: int64

了解预测值的分布

总体分布概况
查看skewness and kurtosis
查看预测值的具体频数
Train_data['label']

Train_data['label']

0        0.0
1        0.0
2        4.0
3        0.0
4        0.0
        ... 
99995    4.0
99996    0.0
99997    0.0
99998    0.0
99999    1.0
Name: label, Length: 100000, dtype: float64

Train_data['label'].value_counts()

Train_data['label'].value_counts()

0.0    58883
4.0    19660
2.0    12994
1.0     6522
3.0     1941
Name: label, dtype: int64

内联代码片

 1) 总体分布概况（无界约翰逊分布等）

import scipy.stats as st
y = Train_data['label']
plt.figure(1); plt.title('Default')
sns.distplot(y, rug=True, bins=20)
plt.figure(2); plt.title('Normal')
sns.distplot(y, kde=False, fit=st.norm)
plt.figure(3); plt.title('Log Normal')
sns.distplot(y, kde=False, fit=st.lognorm)

 2）查看skewness and kurtosis

sns.distplot(Train_data['label']);
print("Skewness: %f" % Train_data['label'].skew())
print("Kurtosis: %f" % Train_data['label'].kurt())

result:
Skewness: 0.871005
Kurtosis: -1.009573

Train_data.skew(), Train_data.kurt()

(id       0.000000
 label    0.917596
 dtype: float64, id      -1.200000
 label   -0.825276
 dtype: float64)

sns.distplot(Train_data.kurt(),color='orange',axlabel ='Kurtness')

result:

 3) 查看预测值的具体频数

plt.hist(Train_data['label'], orientation = 'vertical',histtype = 'bar', color ='red')
plt.show()

result:

用pandas_profiling生成数据报告

import pandas_profiling
pfr = pandas_profiling.ProfileReport(data_train)
pfr.to_file("./example.html")