机器学习实践中的 7 种常见错误

转载 2016年08月30日 19:35:57



Statistical modeling is a lot like engineering.

In engineering, there are various ways to build a key-value storage, and each design makes a different set of assumptions about the usage pattern. In statistical modeling, there are various algorithms to build a classifier, and each algorithm makes a different set of assumptions about the data.

When dealing with small amounts of data, it’s reasonable to try as many algorithms as possible and to pick the best one since the cost of experimentation is low. But as we hit “big data”, it pays off to analyze the data upfront and then design the modeling pipeline (pre-processing, modeling, optimization algorithm, evaluation, productionization) accordingly.

As pointed out in my previous post, there are dozens of ways to solve a given modeling problem. Each model assumes something different, and it’s not obvious how to navigate and identify which assumptions are reasonable. In industry, most practitioners pick the modeling algorithm they are most familiar with rather than pick the one which best suits the data. In this post, I would like to share some common mistakes (the don't-s). I’ll save some of the best practices (the do-s) in a future post.

1. Take default loss function for granted

Many practitioners train and pick the best model using the default loss function (e.g., squared error). In practice, off-the-shelf loss function rarely aligns with the business objective. Take fraud detection as an example. When trying to detect fraudulent transactions, the business objective is to minimize the fraud loss. The off-the-shelf loss function of binary classifiers weighs false positives and false negatives equally. To align with the business objective, the loss function should not only penalize false negatives more than false positives, but also penalize each false negative in proportion to the dollar amount. Also, data sets in fraud detection usually contain highly imbalanced labels. In these cases, bias the loss function in favor of the rare case (e.g., through up/down sampling).

2. Use plain linear models for non-linear interaction

When building a binary classifier, many practitioners immediately jump to logistic regression because it’s simple. But, many also forget that logistic regression is a linear model and the non-linear interaction among predictors need to be encoded manually. Returning to fraud detection, high order interaction features like "billing address = shipping address and transaction amount < $50" are required for good model performance. So one should prefer non-linear models like SVM with kernel or tree based classifiers that bake in higher-order interaction features.

3. Forget about outliers

Outliers are interesting. Depending on the context, they either deserve special attention or should be completely ignored. Take the example of revenue forecasting. If unusual spikes of revenue are observed, it's probably a good idea to pay extra attention to them and figure out what caused the spike. But if the outliers are due to mechanical error, measurement error or anything else that’s not generalizable, it’s a good idea to filter out these outliers before feeding the data to the modeling algorithm.

Some models are more sensitive to outliers than others. For instance, AdaBoost might treat those outliers as "hard" cases and put tremendous weights on outliers while decision tree might simply count each outlier as one false classification. If the data set contains a fair amount of outliers, it's important to either use modeling algorithm robust against outliers or filter the outliers out.

4. Use high variance model when n<<p

SVM is one of the most popular off-the-shelf modeling algorithms and one of its most powerful features is the ability to fit the model with different kernels. SVM kernels can be thought of as a way to automatically combine existing features to form a richer feature space. Since this power feature comes almost for free, most practitioners by default use kernel when training a SVM model. However, when the data has n<<p (number of samples << number of features) --  common in industries like medical data -- the richer feature space implies a much higher risk to overfit the data. In fact, high variance models should be avoided entirely when n<<p.

5. L1/L2/... regularization without standardization

Applying L1 or L2 to penalize large coefficients is a common way to regularize linear or logistic regression. However, many practitioners are not aware of the importance of standardizing features before applying those regularization.

Returning to fraud detection, imagine a linear regression model with a transaction amount feature. Without regularization, if the unit of transaction amount is in dollars, the fitted coefficient is going to be around 100 times larger than the fitted coefficient if the unit were in cents. With regularization, as the L1 / L2 penalize larger coefficient more, the transaction amount will get penalized more if the unit is in dollars. Hence, the regularization is biased and tend to penalize features in smaller scales. To mitigate the problem, standardize all the features and put them on equal footing as a preprocessing step.

6. Use linear model without considering multi-collinear predictors

Imagine building a linear model with two variables X1 and X2 and suppose the ground truth model is Y=X1+X2. Ideally, if the data is observed with small amount of noise, the linear regression solution would recover the ground truth. However, if X1 and X2 are collinear, to most of the optimization algorithms' concerns, Y=2*X1, Y=3*X1-X2 or Y=100*X1-99*X2 are all as good. The problem might not be detrimental as it doesn't bias the estimation. However, it does make the problem ill-conditioned and make the coefficient weight uninterpretable.

7. Interpreting absolute value of coefficients from linear or logistic regression as feature importance

Because many off-the-shelf linear regressor returns p-value for each coefficient, many practitioners believe that for linear models, the bigger the absolute value of the coefficient, the more important the corresponding feature is. This is rarely true as (a) changing the scale of the variable changes the absolute value of the coefficient (b) if features are multi-collinear, coefficients can shift from one feature to others. Also, the more features the data set has, the more likely the features are multi-collinear and the less reliable to interpret the feature importance by coefficients.

So there you go: 7 common mistakes when doing ML in practice. This list is not meant to be exhaustive but merely to provoke the reader to consider modeling assumptions that may not be applicable to the data at hand. To achieve the best model performance, it is important to pick the modeling algorithm that makes the most fitting assumptions -- not just the one you’re most familiar with.






1. 想当然地使用缺省损失函数



当构建一个二元分类器时,很多实践者会立即跳转到逻辑回归,因为它很简单。但是,很多人也忘记了逻辑回归是一种线性模型,预测变量间的非线性交互需要手动编码。回到欺诈检测问题,要获得好的模型性能,像“billing address = shipping address and transaction amount < $50”这种高阶交互特征是必须的。因此,每个人都应该选择适合高阶交互特征的带核SVM或基于树的分类器。









6. 不考虑线性相关直接使用线性模型

设想建立一个具有两变量X1和X2的线性模型,假设真实模型是Y=X1+X2。理想地,如果观测数据含有少量噪声,线性回归解决方案将会恢复真实模型。然而,如果X1和X2线性相关(大多数优化算法所关心的),Y=2*X1, Y=3*X1-X2或Y=100*X1-99*X2都一样好,这一问题可能并无不妥,因为它是无偏估计。然而,它却会使问题变得病态,使系数权重变得无法解释。

7. 将线性或逻辑回归模型的系数绝对值解释为特征重要性





  • foolsnowman
  • foolsnowman
  • 2016年06月08日 18:08
  • 2371


前面看过了CNN的基本结构,经典的模式如下图:   上图经典的CNN模型主要可以概括为三个部分: convolution层:convolution是将原来的输入向量映射成...
  • u010223750
  • u010223750
  • 2016年05月06日 20:41
  • 2147


参考:http://kubicode.me/2015/08/16/Machine%20Learning/Common-Interview/?from=singlemessage# htt...
  • sinat_31337047
  • sinat_31337047
  • 2017年09月09日 09:06
  • 305

数值分析: 病态问题 & 算法稳定

  • Dennis_BIRL
  • Dennis_BIRL
  • 2016年11月26日 13:36
  • 1903

机器学习实践中的 7 种常见错误

转载自:http://blog.jobbole.com/70684/ 编注:本文作者是 Codecademy 的分析主管 Cheng-Tao Chu,其专长是数据挖掘和机器学习,之前在 Go...
  • u013524655
  • u013524655
  • 2014年11月09日 11:57
  • 316


统计建模非常像工程学。 在工程学中,有多种构建键-值存储系统的方式,每个设计都会构造一组不同的关于使用模式的假设集合。在统计建模中,有很多分类器构建算法,每个算法构造一组不同的关于数据的假设集合。 ...
  • u013995172
  • u013995172
  • 2015年04月14日 10:30
  • 303


1.损失函数与风险函数 监督学习的任务就是学习一个模型f作为决策函数,对于给定的输入X,给出相应的输出f(X),这个输出的预测值f(X)与真实值Y可能一致也可能不一致,用一个损失函数(loss fu...
  • u012507022
  • u012507022
  • 2016年11月17日 19:18
  • 3673

吴恩达机器学习笔记(2) 逻辑回归

  • u014202086
  • u014202086
  • 2017年08月09日 10:39
  • 478


今天看了一篇论文:A Few Useful Things to Know about Machine learning 机器学习实践中12条教训 很值得参考借鉴,这些都是大牛总结实践经验!...
  • huruzun
  • huruzun
  • 2014年11月25日 00:37
  • 1105


  • joomlaer
  • joomlaer
  • 2015年03月06日 10:06
  • 1411
您举报文章:机器学习实践中的 7 种常见错误