统计方法进行异常检测_异常检测的统计技术

统计方法进行异常检测

Anomaly and fraud detection is a multi-billion-dollar industry. According to a Nilson Report, the amount of global credit card fraud alone was USD 7.6 billion in 2010. In the UK fraudulent credit card transaction losses were estimated at more than USD 1 billion in 2018. To counter these kinds of financial losses a huge amount of resources are employed to identify frauds and anomalies in every single industry.

异常和欺诈检测是一个价值数十亿美元的行业。 根据尼尔森报告( Nilson Report) ，2010年，仅全球信用卡欺诈的金额就达到76亿美元。在英国，2018年欺诈性信用卡交易损失估计超过10亿美元。 为了应对此类财务损失，在每个行业中都使用大量资源来识别欺诈和异常情况。

In data science, “Outlier”, “Anomaly” and “Fraud” are often synonymously used, but there are subtle differences. An “outliers’ generally refers to a data point that somehow stands out from the rest of the crowd. However, when this outlier is completely unexpected and unexplained, it becomes an anomaly. That is to say, all anomalies are outliers but not necessarily all outliers are anomalies. In this article, however, I am using these terms interchangeably.

在数据科学中，“异常值”，“异常”和“欺诈”经常被同义词使用，但存在细微的差异。 “异常值”通常是指以某种方式在其他人群中脱颖而出的数据点。 但是，当此异常值完全出乎意料且无法解释时，它将成为异常。 也就是说，所有异常均是异常值，但不一定所有异常均是异常值。 但是，在本文中，我可以互换使用这些术语。

There are numerous reasons why understanding and detecting outliers are important. As a data scientist when we make data preparation we take great care in understanding if there is any data point unexplained, which may have entered erroneously. Sometimes we filter completely legitimate outlier data points and remove them to ensure greater model performance.

理解和检测异常值很重要的原因有很多。 作为数据科学家，在进行数据准备时，我们会非常小心地理解是否存在无法解释的任何数据点，这些数据点可能会错误输入。 有时，我们会过滤掉完全合法的异常数据点并删除它们，以确保更高的模型性能。

There is also a huge industrial application of anomaly detection. Credit card fraud detection is the most cited one but in numerous other cases anomaly detection is an essential part of doing business such as detecting network intrusion, identifying instrument fai