《Anomaly Detection-A survey》

《Anomaly Detection-A survey》

Anomaly Detection

检测技术

异常类型

Point Anomaly

数据点的异常，单个指标值，或者多个指标值形成的向量点。

Contextual Anomaly

每个数据点都包含两类属性:

Contextual Attributes: 空间数据的经纬度、时间序列数据的时间戳、每个数据点附带的某些属性

Behavior Attributes: 数据点的实际value值

Collective Anomaly

无法从单个数据点去判断是否异常，而是一段数据序列的集合型异常。

标签与检测技术

normal & anomalous

detection techniques

监督型异常检测：需要打了便签的训练集，根据训练的模型来检测。这部分涉及到异常的插入。

半-监督型异常检测：针对normal建立模型。

无监督型异常检测：

应用场景

结合很多实际的应用场景，简要介绍了各种场景的异常检测方法。

整个第三部分都是在讲这个。

基于分类的异常检测

多分类：多个normal类，不在其中的为异常

一分类：只有一个normal类型，不在其中的为异常

多种方法

• 基于神经网络
• 基于贝叶斯网络
• 基于SVM支持向量机
• 基于规则

优缺点

基于近邻的异常检测

使用距离、相似度作为衡量指标

分成两类：

• 距离
• 相对密度

前K个近邻

*The anomaly score of a data instance is defifined as its distance to its k th nearest neighbor in a given data set

相对密度

Local Outlier Factor (LOF)

Connectivity based Outlier Factor* (COF)

计算复杂度与优缺点

• 复杂度

N的平方

• 优缺点

…

基于聚类的异常检测

同样分成了三类，适用于三种场景。

assumption1 ： Normal data instances belong to a cluster in the data, while anomalies either do not belong to any cluster.

assumption2：Normal data instances lie close to their closest cluster centroid, while anomalies are far away from their closest cluster centroid.

assumption3：Normal data instances belong to large and dense clusters, while anomalies either belong to small or sparse clusters.

聚类与近邻的区别

The key difffference between the two techniques, however, is that clustering based techniques evaluate each instance with respect to the cluster it belongs to, while nearest neighbor based techniques analyze each instance with respect to its local neighborhood.

计算复杂度与优缺点

基于统计的异常检测

基本定律:

The underlying principle of any statistical anomaly detection technique is: “An anomaly is an observation which is suspected of being partially or wholly irrelevant because it is not generated by the stochastic model assumed”。

统计异常检测的假设:

Normal data instances occur in high probability regions of a stochastic model, while anomalies occur in the low probability regions of the stochastic model.

Parametric Techniques

Parametric techniques assume the knowledge of underlying distribution and estimate the parameters from the given data.

高斯模型

回归模型

参数分布的混合

Non-parametric Techniques

Non-parametric techniques do not generally assume knowledge of underlying distribution.

直方图

kernel function

计算复杂度与优缺点

信息论异常检测

Information theoretic techniques analyze the information content of a data set using difffferent information theoretic measures such as Kolomogorov Complexity, entropy, relative entropy, etc.

Assumption:

Anomalies in data induce irregularities in the information content of the data set.

计算复杂度与优缺点

频谱异常检测

Spectral techniques try to find an approximation of the data using a combination of attributes that capture the bulk of variability in the data.

Assumption:

Data can be embedded into a lower dimensional subspace in which normal instances and anomalies appear signifificantly difffferent.

Handling Contextual Anomaly

The anomaly detection techniques discussed in the previous sections primarily focus on detecting point anomalies. In this section, we will discuss anomaly detection techniques that handle contextual anomalies.

Reduction to Point Anomaly Detection Problem

Utilizing the Structure in Data

Handling Collective Anomaly

Handling Sequential Anomaly

Handling Spatial Anomaly

异常检测技术对的相对优缺点

结论