A Comparative Study on Unsupervised Anomaly Detection for Time Series Experiments and Analysis.pdf

明智健悟

186

80页

1次

2023-01-18

免费下载

A Comparative Study on Unsupervised Anomaly Detection for

Time Series: Experiments and Analysis

Yan Zhao

, Liwei Deng

, Xuanhao Chen

, Chenjuan Guo

, Bin Yang

, Tung Kieu

, Feiteng Huang

Torben Bach Pedersen

, Kai Zheng

, Christian S. Jensen

Aalborg University

University of Electronic Science and Technology of China

Huawei Cloud Database Innovation Lab, China

{yanz, cguo, byang, tungkvt, tbp, csj}@cs.aau.dk

{deng_liwei, xhc}@std.uestc.edu.cn, zhengkai@uestc.edu.cn

huangfeiteng@huawei.com

ABSTRACT

The continued digitization of societal processes translates into a

proliferation of time series data that cover applications such as fraud

detection, intrusion detection, and energy management, where anom-

aly detection is often essential to enable reliability and safety. Many

recent studies target anomaly detection for time series data. Indeed,

area of time series anomaly detection is characterized by diverse

data, methods, and evaluation strategies, and comparisons in ex-

isting studies consider only part of this diversity, which makes it

difﬁcult to select the best method for a particular problem setting. To

address this shortcoming, we introduce taxonomies for data, meth-

ods, and evaluation strategies, provide a comprehensive overview of

unsupervised time series anomaly detection using the taxonomies,

and systematically evaluate and compare state-of-the-art traditional

as well as deep learning techniques. In the empirical study using

nine publicly available datasets, we apply the most commonly-used

performance evaluation metrics to typical methods under a fair im-

plementation standard. Based on the structuring offered by the tax-

onomies, we report on empirical studies and provide guidelines, in

the form of comparative tables, for choosing the methods most suit-

able for particular application settings. Finally, we propose research

directions for this dynamic ﬁeld.

1 INTRODUCTION

The continued, society-wide digitization and the accompanying de-

ployment of sensing technologies generate increasingly massive

amounts of time series data. A time series is a sequence of ob-

servations recorded in chronological order. In time series, a small

number of observations may deviate signiﬁcantly from most obser-

vations because they are generated by different processes. Depend-

ing on the setting and application domain, such observations are

called anomalies, abnormalities, deviants, intrusions, outliers, fail-

ures, discordant observations, exceptions, aberrations, peculiarities,

or contaminants [

]. For simplicity, we use the term “anomaly”

throughout the paper.

Anomaly detection for time series has been studied in diverse

settings, such as credit card fraud detection, health care insurance,

intrusion detection in cyber security, and fault detection in safety

critical systems [

]. Anomaly de-

tection methods aim to identify observations that differ signiﬁcantly

from the bulk of observations. Anomaly detection methods may

be supervised or unsupervised. The former require the availability

of labels indicating ground-truth anomaly observations, which is

not the case in many real-world application settings. In contrast,

unsupervised methods, the focus on this paper, do not require the

availability of ground truth anomaly labels for their functioning

and are more generally applicable. In particular, time-consuming

and labor-intensive human labeling is not needed, and it becomes

possible to identify unanticipated anomalies that may have gone

unnoticed during manual labeling.

The wide range of existing unsupervised methods for time series

anomaly detection can be classiﬁed as traditional methods [

] or as deep learning methods [

]. However, a key challenge when attempting to leverage this

body of proposals for performing anomaly detection in real-world

application settings is the lack of guidance as to which methods are

appropriate for use in different settings. In the case of supervised

methods, the labels used offer such guidance, but a comprehensive

mapping of unsupervised methods to application settings is needed.

To achieve this, we face two main challenges.

Challenge I: diversity of data, methods, and evaluation strategies.

It is difﬁcult to select a suitable unsupervised method among the

many alternatives for a particular application because this amounts

to a multi-criteria decision-making problem. First, it is necessary to

take into account the complex and diverse nature of the underlying

time series data, such as the dimensionality, the stationarity, and the

temporal correlations among observations. Second, many unsuper-

vised methods are available, and they use different ways of detecting

anomalies, including density-based clustering [

], similarity

search [

], tree-based partitioning [

], one-class classi-

ﬁcation [

], reconstruction [

], and prediction [

It is non-trivial to select a suitable method simply based on the

description of the method. Third, unlike in time series prediction

and classiﬁcation, where the output is a value that straightforwardly

indicates a future value or a class label, the outputs of time series

anomaly detection methods are diverse and include density values,

distance values, reconstruction errors, etc. Different ways of deriving

anomalies from such values exist, which also yields different strate-

gies for evaluating anomalies. No single evaluation strategy exists

that indicates the suitability of a method for an application. Rather,

it occurs commonly that a method achieves good results according

to one evaluation strategy but performs poorly according to another.

For example, precision and recall, which are used commonly for as-

sessing performance, are often conﬂicting and may be traded for one

another according to application requirements. To illustrate, AIOps

(artiﬁcial intelligence for IT operations) applications require high

precision, as operators do not want to be disturbed by frequent false

arXiv:2209.04635v1 [cs.LG] 10 Sep 2022

Yan Zhao

, Liwei Deng

, Xuanhao Chen

, Chenjuan Guo

, Bin Yang

, Tung Kieu

, Feiteng Huang

, Torben Bach Pedersen

, Kai Zheng

, Christian S. Jensen

Table 1: Survey Comparisons and Contributions

Survey and benchmark Application domain

Supervised/

Unsupervised

Taxonomy

of data

Taxonomy

of methods

Taxonomy of

evaluation

Traditional

methods

Deep learning

methods

Empirical

study

Method

recommendation

2003 [30] Network intrusion Unsupervised ×

√

2004 [15] General anomaly Both ×

√

√ √

× ×

2009 [4] General anomaly Both ×

√

√ √

× ×

2013 [14] Temporal data anomaly Both ×

√

√ √

× ×

2017 [6] Streaming data anomaly Unsupervised ×

√

√ √

2018 [24] Video anomaly Unsupervised ×

√

× ×

√ √

2019 [3] General anomaly Both

√ √

× ×

√

× ×

2019 [28] Network anomaly Both ×

√

× ×

√

× ×

2020 [18] Time series anomaly Supervised × × × ×

√ √

2020 [40] General anomaly Supervised

√ √

× ×

√

× ×

This work Time series anomaly Unsupervised

√ √ √ √ √ √ √

alarms, while applications in intensive care units that monitor signs

of life (e.g., respiratory rate and blood pressure) prioritize recall at

the cost of precision [32].

Challenge II: fair comparison. The effectiveness of unsupervised

methods depends highly on the settings of hyperparameters, which

increases the difﬁculty of selecting a suitable method. As unsuper-

vised methods train models without labels, no fair or golden standard

exists for choosing suitable hyperparameter settings that maximize

performance. In other words, it is challenging to select hyperparame-

ters fairly and effectively in an unsupervised manner. This increases

the need for being able to compare unsupervised methods fairly in

different settings to understand thoroughly the advantages and dis-

advantages of these methods, such that their suitability for a speciﬁc

application could be well demonstrated.

Therefore, as more and more methods become available, there

is an increasing need for a comprehensive framework and study of

the suitability of unsupervised time series anomaly detection meth-

ods for different application settings. Table 1 offers an overview

of prior surveys and benchmarks, showing that these focus mostly

on general anomaly detection, e.g., detecting novelties in images

and identifying novel molecular structures in pharmaceutical re-

search [

] or a particular application domain (e.g., video

anomaly) [

], instead of time series anomaly de-

tection. In addition, most of these existing studies do not provide

comparative taxonomies for data, methods, and evaluation strategies,

or they target only speciﬁc types of traditional or deep learning meth-

ods. Furthermore, there is an unmet need for a comparative empirical

study of the ﬁt of methods for given data and application settings.

The only exception is the study of Choudhary et al. [

] that targets

streaming data anomaly detection rather than time series anomaly

detection. In addition, it only selects a suitable method based on

attributes exhibited by datasets from different application domains

and latency requirements. It disregards general data attributes, e.g.,

dimensionality and stationarity, and evaluation strategies.

This work aims to address the above challenges by providing a

comprehensive reference for researchers and practitioners seek to

utilize unsupervised time series anomaly detection techniques in

speciﬁc applications. In particular, this study focuses on providing

guidance on which unsupervised time series anomaly detection meth-

ods in different settings, and attempts to explore how to employ the

methods on different data. Speciﬁcally, the paper makes three key

contributions.

Qualitative Analysis.

This paper offers a comprehensive overview

of unsupervised time series anomaly detection by proposing tax-

onomies for data, methods, and evaluation strategies.

Quantitative Analysis.

The paper considers

carefully se-

lected unsupervised anomaly detection methods, including repre-

sentative traditional and deep learning methods, and evaluates them

using nine publicly available datasets in terms of accuracy, robust-

ness, and running time efﬁciency while ensuring consistent main

hyperparameter settings to ensure fairness. We provide a comprehen-

sive overview of the results. The related source code can be accessed

at Github

Guidelines of Method Recommendation.

The experimental

results and accompanying analyses serve as guidelines to suggest

an appropriate method for a particular dataset. Speciﬁcally, we of-

fer several comparative tables that provide recommendations of

techniques suitable for the applications at hand based on the pro-

posed taxonomies and experimental evaluations. This paper can be

used as a hands-on guide for understanding, using, or developing

unsupervised anomaly detection methods for different real-world

applications.

The remainder of the paper is structured as follows. We deﬁne

taxonomies for time series anomaly detection data, methods, and

evaluation strategies in Sec. 2. Sec. 3 details the experimental setup,

followed by Sec. 4 that gives the experimental results and offers

method recommendations. We ﬁnally conclude the paper and provide

research directions in Sec. 5.

2 TAXONOMIES

It is a multi-criteria decision-making problem to choose the best

unsupervised time series anomaly detection method for a particular

application, where the key criteria include the anomaly detection

data, methods, and evaluation strategies. Therefore, we provide three

taxonomies for the data, methods, and evaluation strategies, thus

achieving a structured basis for method recommendation.

2.1 Taxonomy of Data

Challenges in time series anomaly detection method recommenda-

tion in terms of data include diverse dimensionality and stationarity,

based on which we propose a data taxonomy.

2.1.1 Univariate vs. Multivariate. The dimensionality is the num-

ber of features captured in each observation [

]. A time series

𝑇

a sequence of

𝑘

-dimensional vectors

𝑇 = ⟨𝑆

, 𝑆

, ..., 𝑆

𝑛

⟩

, where

https://github.com/yan20191113/ADTS

of 80

免费下载

关注

评论