Scikit Learn-异常检测 (Scikit Learn - Anomaly Detection)

Here, we will learn about what is anomaly detection in Sklearn and how it is used in identification of the data points.

在这里，我们将了解什么是Sklearn中的异常检测以及如何将其用于识别数据点。

Anomaly detection is a technique used to identify data points in dataset that does not fit well with the rest of the data. It has many applications in business such as fraud detection, intrusion detection, system health monitoring, surveillance, and predictive maintenance. Anomalies, which are also called outlier, can be divided into following three categories −

异常检测是一种用于识别数据集中与其他数据不太吻合的数据点的技术。 它在商业中具有许多应用程序，例如欺诈检测，入侵检测，系统运行状况监视，监视和预测性维护。 异常也称为离群值，可以分为以下三类：

• Point anomalies − It occurs when an individual data instance is considered as anomalous w.r.t the rest of the data.

  点异常 -当单个数据实例被认为与其余数据异常时，会发生异常。

• Contextual anomalies − Such kind of anomaly is context specific. It occurs if a data instance is anomalous in a specific context.

  上下文异常 -这种异常是上下文特定的。 如果数据实例在特定上下文中异常，则会发生这种情况。

• Collective anomalies − It occurs when a collection of related data instances is anomalous w.r.t entire dataset rather than individual values.

  集体异常 -当相关数据实例的集合相对于整个数据集而不是单个值异常时，就会发生这种情况。

方法 (Methods)

Two methods namely outlier detection and novelty detection can be used for anomaly detection. It’s necessary to see the distinction between them.

异常检测可以使用异常检测和新颖性检测这两种方法。 有必要看到它们之间的区别。

离群值检测 (Outlier detection)

The training data contains outliers that are far from the rest of the data. Such outliers are defined as observations. That’s the reason, outlier detection estimators always try to fit the region having most concentrated training data while ignoring the deviant observations. It is also known as unsupervised anomaly detection.

训练数据包含离其他数据远的异常值。 这些异常值被定义为观察值。 这就是原因，离群检测估计器总是尝试拟合训练数据最集中的区域，而忽略了异常观测值。 这也称为无监督异常检测。

新颖性检测 (Novelty detection)

It is concerned with detecting an unobserved pattern in new observations which is not included in training data. Here, the training data is not polluted by the outliers. It is also known as semi-supervised anomaly detection.

它与在训练数据中不包括的新观察中检测到未观察到的模式有关。 在这里，训练数据不受异常值的污染。 这也称为半监督异常检测。

There are set of ML tools, provided by scikit-learn, which can be used for both outlier detection as well novelty detection. These tools first implementing object learning from the data in an unsupervised by using fit () method as follows −

scikit-learn提供了一套ML工具，可用于异常检测和新颖性检测。 这些工具首先通过使用fit()方法在无监督的情况下从数据中实现对象学习-

estimator.fit(X_train)

Now, the new observations would be sorted as inliers (labeled 1) or outliers (labeled -1) by using predict() method as follows −

现在，可以通过使用predict()方法将新观察值分类为离群值(标记 为1)或离群值(标记为-1) ，如下所示：

estimator.fit(X_test)

The estimator will first compute the raw scoring function and then predict method will make use of threshold on that raw scoring function. We can access this raw scoring function with the help of score_sample method and can control the threshold by contamination parameter.

估计器将首先计算原始评分函数，然后预测方法将使用该原始评分函数的阈值。 我们可以借助score_sample方法访问此原始评分功能，并可以通过污染参数控制阈值。

We can also define decision_function method that defines outliers as negative value and inliers as non-negative value.

我们还可以定义Decision_function方法，将离群值定义为负值，将离群值定义为非负值。

estimator.decision_function(X_test)

用于异常值检测的Sklearn算法 (Sklearn algorithms for Outlier Detection)

Let us begin by understanding what an elliptic envelop is.

让我们首先了解什么是椭圆形信封。

拟合椭圆形信封 (Fitting an elliptic envelop)

This algorithm assume that regular data comes from a known distribution such as Gaussian distribution. For outlier detection, Scikit-learn provides an object named covariance.EllipticEnvelop.

该算法假定常规数据来自已知分布，例如高斯分布。 为了检测异常值，Scikit-learn提供了一个名为covariance.EllipticEnvelop的对象。

This object fits a robust covariance estimate to the data, and thus, fits an ellipse to the central data points. It ignores the points outside the central mode.

该对象将稳健的协方差估计值拟合到数据，因此将椭圆拟合到中心数据点。 它忽略中心模式之外的点。

参量 (Parameters)

Following table consist the parameters used by sklearn. covariance.EllipticEnvelop method −

下表包含sklearn使用的参数。 covariance.EllipticEnvelop方法-

属性 (Attributes)

Following table consist the attributes used by sklearn. covariance.EllipticEnvelop method −

下表包含sklearn使用的属性。 covariance.EllipticEnvelop方法-

Implementation Example

实施实例

import numpy as np^M
from sklearn.covariance import EllipticEnvelope^M
true_cov = np.array([[.5, .6],[.6, .4]])
X = np.random.RandomState(0).multivariate_normal(mean = [0, 0], cov=true_cov,size=500)
cov = EllipticEnvelope(random_state = 0).fit(X)^M
# Now we can use predict method. It will return 1 for an inlier and -1 for an outlier.
cov.predict([[0, 0],[2, 2]])

Output

输出量

array([ 1, -1])

隔离林 (Isolation Forest)

In case of high-dimensional dataset, one efficient way for outlier detection is to use random forests. The scikit-learn provides ensemble.IsolationForest method that isolates the observations by randomly selecting a feature. Afterwards, it randomly selects a value between the maximum and minimum values of the selected features.

对于高维数据集，一种有效的离群值检测方法是使用随机森林。 scikit-learn提供了ensemble.IsolationForest方法，该方法通过随机选择特征来隔离观察结果。 之后，它会在所选特征的最大值和最小值之间随机选择一个值。

Here, the number of splitting needed to isolate a sample is equivalent to path length from the root node to the terminating node.

在这里，隔离样本所需的拆分次数等于从根节点到终止节点的路径长度。

参量 (Parameters)

Followings table consist the parameters used by sklearn. ensemble.IsolationForest method −

跟随表包括sklearn使用的参数。 ensemble.IsolationForest方法-

属性 (Attributes)

Following table consist the attributes used by sklearn. ensemble.IsolationForest method −

下表包含sklearn使用的属性。 ensemble.IsolationForest方法-

Implementation Example

实施实例

The Python script below will use sklearn. ensemble.IsolationForest method to fit 10 trees on given data

下面的Python脚本将使用sklearn。 ensemble.IsolationForest方法可在给定数据上拟合10棵树

from sklearn.ensemble import IsolationForest
import numpy as np
X = np.array([[-1, -2], [-3, -3], [-3, -4], [0, 0], [-50, 60]])
OUTDClf = IsolationForest(n_estimators = 10)
OUTDclf.fit(X)

Output

输出量

IsolationForest(
behaviour = 'old', bootstrap = False, contamination='legacy',
max_features = 1.0, max_samples = 'auto', n_estimators = 10, n_jobs=None,
random_state = None, verbose = 0
)

局部离群因子 (Local Outlier Factor)

Local Outlier Factor (LOF) algorithm is another efficient algorithm to perform outlier detection on high dimension data. The scikit-learn provides neighbors.LocalOutlierFactor method that computes a score, called local outlier factor, reflecting the degree of anomality of the observations. The main logic of this algorithm is to detect the samples that have a substantially lower density than its neighbors. Thats why it measures the local density deviation of given data points w.r.t. their neighbors.

局部离群因子(LOF)算法是对高维数据执行离群检测的另一种有效算法。 scikit-learn提供neighbors.LocalOutlierFactor方法，该方法计算得分(称为局部异常值)，以反映观测值的异常程度。 该算法的主要逻辑是检测密度远低于其邻居密度的样本。 这就是为什么它测量给定数据点及其邻居的局部密度偏差的原因。

参量 (Parameters)

Followings table consist the parameters used by sklearn. neighbors.LocalOutlierFactor method

跟随表包括sklearn使用的参数。 neighbors.LocalOutlierFactor方法

属性 (Attributes)

Following table consist the attributes used by sklearn.neighbors.LocalOutlierFactor method −

下表包含sklearn.neighbors.LocalOutlierFactor方法使用的属性-

Implementation Example

实施实例

The Python script given below will use sklearn.neighbors.LocalOutlierFactor method to construct NeighborsClassifier class from any array corresponding our data set

下面给出的Python脚本将使用sklearn.neighbors.LocalOutlierFactor方法从对应于我们数据集的任何数组构造NeighborsClassifier类

from sklearn.neighbors import NearestNeighbors
samples = [[0., 0., 0.], [0., .5, 0.], [1., 1., .5]]
LOFneigh = NearestNeighbors(n_neighbors = 1, algorithm = "ball_tree",p=1)
LOFneigh.fit(samples)

Output

输出量

NearestNeighbors(
algorithm = 'ball_tree', leaf_size = 30, metric='minkowski',
metric_params = None, n_jobs = None, n_neighbors = 1, p = 1, radius = 1.0
)

Example

例

Now, we can ask from this constructed classifier is the closet point to [0.5, 1., 1.5] by using the following python script −

现在，我们可以使用以下python脚本从此构造的分类器中询问[0.5，1.，1.5]的壁橱点-

print(neigh.kneighbors([[.5, 1., 1.5]])

Output

输出量

(array([[1.7]]), array([[1]], dtype = int64))

一类SVM (One-Class SVM)

The One-Class SVM, introduced by Schölkopf et al., is the unsupervised Outlier Detection. It is also very efficient in high-dimensional data and estimates the support of a high-dimensional distribution. It is implemented in the Support Vector Machines module in the Sklearn.svm.OneClassSVM object. For defining a frontier, it requires a kernel (mostly used is RBF) and a scalar parameter.

Schölkopf等人介绍的One-Class SVM是无监督的离群值检测。 它在高维数据中也非常有效，并估计了高维分布的支持。 它在Sklearn.svm.OneClassSVM对象的“ 支持向量机”模块中实现。 为了定义边界，它需要一个内核(最常用的是RBF)和一个标量参数。

For better understanding let's fit our data with svm.OneClassSVM object −

为了更好地理解，让我们将数据与svm.OneClassSVM对象配合起来 -

例 (Example)

from sklearn.svm import OneClassSVM
X = [[0], [0.89], [0.90], [0.91], [1]]
OSVMclf = OneClassSVM(gamma = 'scale').fit(X)

Now, we can get the score_samples for input data as follows −

现在，我们可以获得输入数据的score_samples，如下所示：

OSVMclf.score_samples(X)

输出量 (Output)

array([1.12218594, 1.58645126, 1.58673086, 1.58645127, 1.55713767])

翻译自: https://www.tutorialspoint.com/scikit_learn/scikit_learn_anomaly_detection.htm