Non-stationary Transformers.pdf

smith0907

115

21页

0次

2023-10-22

50墨值下载

Non-stationary Transformers:

Exploring the Stationarity in Time Series Forecasting

Yong Liu

∗

, Haixu Wu

∗

, Jianmin Wang, Mingsheng Long

School of Software, BNRist, Tsinghua University, China

{liuyong21,whx20}@mails.tsinghua.edu.cn, {jimwang,mingsheng}@tsinghua.edu.cn

Abstract

Transformers have shown great power in time series forecasting due to their

global-range modeling ability. However, their performance can degenerate ter-

ribly on non-stationary real-world data in which the joint distribution changes

over time. Previous studies primarily adopt stationarization to attenuate the non-

stationarity of original series for better predictability. But the stationarized series

deprived of inherent non-stationarity can be less instructive for real-world bursty

events forecasting. This problem, termed over-stationarization in this paper, leads

Transformers to generate indistinguishable temporal attentions for different series

and impedes the predictive capability of deep models. To tackle the dilemma

between series predictability and model capability, we propose Non-stationary

Transformers as a generic framework with two interdependent modules: Series

Stationarization and De-stationary Attention. Concretely, Series Stationarization

uniﬁes the statistics of each input and converts the output with restored statis-

tics for better predictability. To address the over-stationarization problem, De-

stationary Attention is devised to recover the intrinsic non-stationary information

into temporal dependencies by approximating distinguishable attentions learned

from raw series. Our Non-stationary Transformers framework consistently boosts

mainstream Transformers by a large margin, which reduces MSE by 49.43% on

Transformer, 47.34% on Informer, and 46.89% on Reformer, making them the

state-of-the-art in time series forecasting. Code is available at this repository:

https://github.com/thuml/Nonstationary_Transformers.

1 Introduction

Time series forecasting has become increasingly ubiquitous in real-world applications, such as weather

forecasting, energy consumption planning, and ﬁnancial risk assessment. Recently, Transformers [

]

have achieved progressive breakthrough on extensive areas [

]. Especially in time

series forecasting, credited to their stacked structure and the capability of attention mechanisms,

Transformers can naturally capture the temporal dependencies from deep multi-level features [

22, 37], thereby ﬁtting the series forecasting task perfectly.

Despite the remarkable architectural design, it is still challenging for Transformers to predict real-

world time series because of the non-stationarity of data. Non-stationary time series is characterized

by the continuous change of statistical properties and joint distribution over time, which makes

the time series less predictable [

]. Besides, it is a fundamental problem to make deep models

generalize well on a varying distribution [

]. In previous work, it is generally acknowledged

to pre-process the time series by stationarization [

], which can attenuate the non-stationarity

of raw time series for better predictability and provide more stable data distribution for deep models.

∗

Equal Contribution

36th Conference on Neural Information Processing Systems (NeurIPS 2022).

arXiv:2205.14415v3 [cs.LG] 18 Oct 2022

Ⅰ Ⅱ Ⅲ

Non-stationary

Varied !, #

(a) Vanilla

Transformer

(b) Transformer with

Series Stationarization

Transformer

Ⅰ

, #

Ⅱ

, #

Ⅲ

, #

Learned

Attention

Zoom in

Figure 1: Visualization of learned temporal attentions for different series with varied mean

and

standard deviation

. (a) is from the vanilla Transformer [

] trained on raw series. (b) is from

the Transformer trained on stationarized series, which presents similar attentions. (c) is from Non-

stationary Transformers, which involves De-stationary Attention to avoid over-stationarization.

However, non-stationarity is the inherent property of real-world time series and also good guidance for

discovering temporal dependencies for forecasting. Experimentally, we observe that training on the

stationarized series will undermine the distinction of attentions learned by Transformers. While vanilla

Transformers [

] can capture distinct temporal dependencies from different series in Figure 1(a),

Transformers trained on the stationarized series tend to generate indistinguishable attentions in

Figure 1(b). This problem, named by the over-stationarization, will bring unexpected side-effect

that makes Transformers fail to capture eventful temporal dependencies, limit the model’s predictive

ability, and even induce the model to generate outputs with huge non-stationarity deviation from the

ground truth. Thus, how to attenuate time series non-stationarity towards better predictability and

mitigate the over-stationarization problem for model capability simultaneously is the key problem to

further improve the performance of forecasting.

In this paper, we explore the effect of stationarization in time series forecasting and propose Non-

stationary Transformers as a general framework, which empowers Transformer [

] and its efﬁcient

variants [

] with great predictive ability for real-world time series. The proposed framework

involves two interdependent modules: Series Stationarization to increase the predictability of non-

stationary series and De-stationary Attention to alleviate over-stationarization. Technically, Series

Stationarization adopts a simple but effective normalization strategy to unify the key statistics of

each series without extra parameters. And De-stationary Attention approximates the attention of

unstationarized data and compensates the intrinsic non-stationarity of raw series. Beneﬁting from

the above designs, Non-stationary Transformers can take advantage of the great predictability of

stationarized series and crucial temporal dependencies discovered from original non-stationary data.

Our method achieves state-of-the-art performance on six real-world benchmarks and can generalize

to various Transformers for further improvement. The contributions lie in three folds:

•

We reﬁne that the predictive capability of non-stationary series is essential in real-world

forecasting. By detailed analysis, we ﬁnd out that current stationarization approaches will

lead to the over-stationarization problem, limiting the predictive capability of Transformers.

•

We propose Non-stationary Transformers as a generic framework, including Series Sta-

tionarization to make the series more predictable and De-stationary Attention to avoid the

over-stationarization problem by re-incorporating the non-stationarity of original series.

•

Non-stationary Transformers consistently boosts four mainstream Transformers by a large

margin and achieves state-of-the-art performance on six real-world benchmarks.

2 Related Work

2.1 Deep Models for Time Series Forecasting

In recent years, deep models with elaboratively designed architectures have achieved great progress

in time series forecasting. RNN-based models [

] are proposed for application in an

of 21

50墨值下载

non-stationary transformers

关注

评论