数据预处理
(一)去极值
1. MAD(Median Absolute Deviation, 绝对值差中位数法)
步骤:
(1)计算因子数据的中位数 X m e d i a n X_{median} Xmedian.
(2)计算因子数据与中位数的绝对偏差值 ∣ X i − X m e d i a n ∣ |X_i-X_{median}| ∣Xi−Xmedian∣.
(3)计算绝对偏差值 ∣ X i − X m e d i a n ∣ |X_i-X_{median}| ∣Xi−Xmedian∣的中位数 M A D MAD MAD.
(4)确定参数 n ( 一 般 来 说 , n 的 值 为 5 ) n(一般来说,n的值为5) n(一般来说,n的值为5),从而确定相应的范围 [ X m e d i a n − n ∗ M A D , X m e d i a n + n ∗ M A D ] [X_{median}-n*MAD,X_{median}+n*MAD] [Xmedian−n∗MAD,Xmedian+n∗MAD].
对超出范围的值进行如下处理:X i = { X m e d i a n + n ∗ M A D X i > X m e d i a n + n ∗ M A D X m e d i a n − n ∗ M A D X i < X m e d i a n − n ∗ M A D X i X m e d i a n − n ∗ M A D < X i < X m e d i a n + n ∗ M A D X_i=\left\{ \begin{array}{ll} X_{median}+n*MAD & X_i > X_{median} + n*MAD\\ X_{median}-n*MAD & X_i < X_{median} - n*MAD\\ X_{i} & X_{median} - n*MAD < X_i < X_{median} + n*MAD\\ \end{array} \right. Xi=⎩⎨⎧Xmedian+n∗MADXmedian−n∗MADXiXi>Xmedian+n∗MADXi<Xmedian−n∗MADXmedian−n∗MAD<Xi<Xmedian+n∗MAD
CODE:
import tushare import numpy as np import pandas as pd pro = tushare.pro_api("api_key") data = pro.daily(ts_code="300224.SZ",start_date="20190101",end_date="20200325") def MAD(series,n): '''MAD(Median Absolute Deviation, 绝对值差中位数法)''' # 计算因子数据的中位数 median = series.quantile(0.5) # 计算每个因子数据与中位数的绝对值偏差|X_i-X_median|的中位数 diff_median = ((series-median).abs()).quantile(0.5) # 确定参数n,从而确定合理的范围[min_range,max_range] max_range = median + n * diff_median min_range = median - n * diff_median # 如果因子数据中的值value<min_range,则value=min_range # 如果因子数据中的值value>max_range,则value=max_range MAD_Series = np.clip(series,min_range,max_range) return MAD_Series MAD = MAD(data.close,5) MAD
2.标准差法
步骤:
(1)计算因子数据的标准差和均值.
(2)确认参数 n n n.
(3)确认范围 [ X m e a n − n ∗ σ , X m e a n + n ∗ σ ] [X_{mean}-n*\sigma,X_{mean}+n*\sigma] [Xmean−n∗σ,Xmean+n∗σ]
(4)对超出范围的值进行如下处理:
X i = { X m e d i a n + n ∗ σ X i > X m e d i a n + n ∗ σ X m e d i a n − n ∗ σ X i < X m e d i a n − n ∗ σ X i X m e d i a n − n ∗ σ < X i < X m e d i a n + n ∗ σ X_i=\left\{ \begin{array}{ll} X_{median}+n*\sigma & X_i > X_{median} + n*\sigma\\ X_{median}-n*\sigma & X_i < X_{median} - n*\sigma\\ X_{i} & X_{median} - n*\sigma < X_i < X_{median} + n*\sigma\\ \end{array} \right. Xi=⎩⎨⎧Xmedian+n∗σXmedian−n∗σXiXi>Xmedian+n∗σXi<Xmedian−n∗σXmedian−n∗σ<Xi<Xmedian+n∗σ
CODE:
import tushare import numpy as np import pandas as pd pro = tushare.pro_api("api_key") data =pro.daily(ts_code="300224.SZ",start_date="20190101",end_date="20200325") def three_sigma(series,n=3): '''三倍标准差去极值法''' # 计算均值 mean = data.close.mean() # 计算标准差 stantard_deviation =data.close.std() # 确定参数n,从而确定合理的范围[min_range,max_range] min_range = mean - n * stantard_deviation max_range = mean + n * stantard_deviation # 如果因子数据中的值value<min_range,则value=min_range # 如果因子数据中的值value>max_range,则value=max_range three_sigma_series = np.clip(series,min_range,max_range) return three_sigma_series three_sigma = three_sigma(data.close,3) three_sigma
3.分位数法
步骤:
(1)确定分位数 n % 和 m % n\%和m\% n%和m%.
(2)确定分位数对应的值 [ m i n _ r a n g e , m a x _ r a n g e ] [min\_range,max\_range] [min_range,max_range].
(3)将范围外的值进行如下处理:
X i = { m a x _ r a n g e X i > m a x _ r a n g e m i n _ r a n g e X i < m i n _ r a n g e X i m i n _ r a n g e < X i < m a x _ r a n g e X_i=\left\{ \begin{array}{ll} max\_range & X_i > max\_range\\ min\_range & X_i < min\_range\\ X_{i} & min\_range < X_i < max\_range\\ \end{array} \right. Xi=⎩⎨⎧max_rangemin_rangeXiXi>max_rangeXi<min_rangemin_range<Xi<max_range
CODE:
import tushare import numpy as np import pandas as pd pro = tushare.pro_api("api_key") data =pro.daily(ts_code="300224.SZ",start_date="20190101",end_date="20200325") def Winsorize(series, min=0.05, max=0.95): '''分位数法''' # 将因子数据升序排列 series = series.sort_values() # 分别计算5%分位数和95%分位数的值 quantile_value = series.quantile([min, max]) # 5%分位数的值 min_range = quantile_value.iloc[0] # 95%分位数的值 max_range = quantile_value.iloc[-1] winsorize_Series = np.clip(series, min_range, max_range) return winsorize_Series winsorize =Winsorize(data.close) winsorize
(二)标准化
1.Z_Score标准化
步骤:
(1)计算因子数据均值 X m e a n X_{mean} Xmean .
(2)计算因子数据标准差 X s t d X_{std} Xstd .
(3)公式:
X i n o r m = X i − X m e a n X s t d X_i^{norm}=\frac{X_i-X_{mean}}{X_{std}} Xinorm=XstdXi−Xmean
CODE:
import tushare import numpy as np import pandas as pd pro = tushare.pro_api("api_key") data =pro.daily(ts_code="300224.SZ",start_date="20190101",end_date="20200325") def Z_Score(series): mean = series.mean() standstard_deviation = series.std() z_score = (series-mean)/standstard_deviation return z_score z_score = Z_Score(data.close) z_score