Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time.pdf

smith0907

719

11页

0次

2023-10-12

25墨值下载

Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time

Series Forecasting

Kashif Rasul

Calvin Seward

Ingmar Schuster

Roland Vollgraf

Abstract

In this work, we propose

TimeGrad

, an autore-

gressive model for multivariate probabilistic time

series forecasting which samples from the data

distribution at each time step by estimating its gra-

dient. To this end, we use diffusion probabilistic

models, a class of latent variable models closely

connected to score matching and energy-based

methods. Our model learns gradients by optimiz-

ing a variational bound on the data likelihood and

at inference time converts white noise into a sam-

ple of the distribution of interest through a Markov

chain using Langevin sampling. We demonstrate

experimentally that the proposed autoregressive

denoising diffusion model is the new state-of-the-

art multivariate probabilistic forecasting method

on real-world data sets with thousands of corre-

lated dimensions. We hope that this method is a

useful tool for practitioners and lays the founda-

tion for future research in this area.

1. Introduction

Classical time series forecasting methods such as those

in (Hyndman & Athanasopoulos, 2018) typically provide

univariate point forecasts, require hand-tuned features to

model seasonality, and are trained individually on each time

series. Deep learning based time series models (Benidis

et al., 2020) are popular alternatives due to their end-to-end

training of a global model, ease of incorporating exogenous

covariates, and automatic feature extraction abilities. The

task of modeling uncertainties is of vital importance for

downstream problems that use these forecasts for (business)

decision making. More often the individual time series

for a problem data set are statistically dependent on each

other. Ideally, deep learning models need to incorporate

this inductive bias in the form of multivariate (Tsay, 2014)

probabilistic methods to provide accurate forecasts.

Zalando Research, M

uhlenstraße 25, 10243 Berlin, Germany.

Correspondence to: Kashif Rasul <kashif.rasul@zalando.de>.

To model the full predictive distribution, methods typically

resort to tractable distribution classes or some type of low-

rank approximations, regardless of the true data distribution.

To model the distribution in a general fashion, one needs

probabilistic methods with tractable likelihoods. Till now

several deep learning methods have been proposed for this

purpose such as autoregressive (van den Oord et al., 2016c)

or generative ones based on normalizing ﬂows (Papamakar-

ios et al., 2019) which can learn ﬂexible models of high

dimensional multivariate time series. Even if the full likeli-

hood is not be tractable, one can often optimize a tractable

lower bound to the likelihood. But still, these methods re-

quire a certain structure in the functional approximators, for

example on the determinant of the Jacobian (Dinh et al.,

2017) for normalizing ﬂows. Energy-based models (EBM)

(Hinton, 2002; LeCun et al., 2006) on the other hand have

a much less restrictive functional form. They approximate

the unnormalized log-probability so that density estimation

reduces to a non-linear regression problem. EBMs have

been shown to perform well in learning high dimensional

data distributions at the cost of being difﬁcult to train (Song

& Kingma, 2021).

In this work, we propose autoregressive EBMs to solve the

multivariate probabilistic time series forecasting problem

via a model we call

TimeGrad

and show that not only are

we able to train such a model with all the inductive biases

of probabilistic time series forecasting, but this model per-

forms exceptionally well when compared to other modern

methods. This autoregressive-EBM combination retains the

power of autoregressive models, such as good performance

in extrapolation into the future, with the ﬂexibility of EBMs

as a general purpose high-dimensional distribution model,

while remaining computationally tractable.

The paper is organized as follows. In Section 2 we ﬁrst set

up the notation and detail the EBM of (Ho et al., 2020)

which forms the basis of our per time-step distribution

model. Section 3 introduces the multivariate probabilis-

tic time series problem and we detail the

TimeGrad

model.

The experiments with extensive results are detailed in Sec-

tion 4. We cover related work in Section 5 and conclude

with some discussion in Section 6.

arXiv:2101.12072v2 [cs.LG] 2 Feb 2021

Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting

2. Diffusion Probabilistic Model

Let

∼ q

)

denote the multivariate training vector

from some input space

X = R

and let

)

denote the

probability density function (PDF) which aims to approxi-

mate

)

and allows for easy sampling. Diffusion mod-

els (Sohl-Dickstein et al., 2015) are latent variable models of

the form

) :=

0:N

) dx

1:N

, where

, . . . , x

are latents of dimension

. Unlike in variational autoen-

coders (Kingma & Welling, 2019) the approximate posterior

q(x

1:N

q(x

1:N

) = Π

n=1

q(x

n−1

)

is not trainable but ﬁxed to a Markov chain (called the

forward process) that gradually adds Gaussian noise to the

signal:

q(x

n−1

) := N(x

;

1 − β

n−1

, β

I).

The forward process uses an increasing variance sched-

ule

, . . . , β

with

∈ (0, 1)

. The joint distribution

0:N

)

is called the reverse process, and is deﬁned as

a Markov chain with learned Gaussian transitions starting

with

p(x

) = N(x

; 0, I)

, where each subsequent transi-

tion of

0:N

) := p(x

)Π

n=N

n−1

)

is given by a parametrization of our choosing denoted by

n−1

) := N(x

n−1

; µ

, n), Σ

, n)I), (1)

with shared parameters

. Both

: R

× N → R

and

: R

× N → R

take two inputs, namely the variable

∈ R

as well as the noise index

n ∈ N

. The goal of

n−1

)

is to eliminate the Gaussian noise added in

the diffusion process. The parameters

are learned to ﬁt

the data distribution

)

by minimizing the negative log-

likelihood via a variational bound using Jensen’s inequality:

min

q(x

)

[−log p

)] ≤

min

q(x

0:N

)

[−log p

0:N

) + log q(x

1:N

)].

This upper bound can be shown to be equal to

min

q(x

0:N

)

−log p(x

) −

n=1

log

n−1

)

q(x

n−1

)

(2)

As shown by (Ho et al., 2020), a property of the forward

process is that it admits sampling

at any arbitrary noise

level

in closed form, since if

:= 1 − β

and

¯α

i=1

its cumulative product, we have:

q(x

) = N(x

;

√

¯α

, (1 − ¯α

)I). (3)

By using the fact that these processes are Markov chains,

the objective in (2) can be written as the KL-divergence

between Gaussian distributions:

− log p

) + D

(q(x

)||p(x

))

n=2

(q(x

n−1

, x

)||p

n−1

)), (4)

and (Ho et al., 2020) shows that by the property (3) the

forward process posterior in these KL divergences when

conditioned on

, i.e.

q(x

n−1

, x

)

are tractable given

q(x

n−1

, x

) = N(x

n−1

; ˜µ

, x

I),

where

˜µ

, x

) :=

√

¯α

n−1

1 − ¯α

√

(1 − ¯α

n−1

)

1 − ¯α

and

1 − ¯α

n−1

1 − ¯α

. (5)

Further, (Ho et al., 2020) shows that the KL-divergence

between Gaussians can be written as:

(q(x

n−1

, x

)||p

n−1

)) =



2Σ

k˜µ

, x

) − µ

, n)k



+ C, (6)

where

is a constant which does not depend on

. So

instead of a parametrization (1) of

that predicts

˜µ

, one can

instead use the property (3) to write

, ) =

√

¯α

√

1 − ¯α



for

 ∼ N(0, I)

and the formula for

˜µ

to obtain

that

must predict

−β

/

√

1 − ¯α

√

, but since

is available to the network, we can choose:

, n) =

√



−

√

1 − ¯α



, n)



where



is a network which predicts

 ∼ N(0, I)

from

so that the objective simpliﬁes to:

,



2Σ

(1 − ¯α

)

k − 

(

√

¯α

√

1 − ¯α

, n)k



(7)

resembling the loss in Noise Conditional Score Networks

(Song & Ermon, 2019; 2020) using score matching. Once

trained, to sample from the reverse process

n−1

∼

n−1

) (1) we can compute

n−1

√



−

√

1 − ¯α



, n)



where

z ∼ N(0, I)

for

n = N, . . . , 2

and

z = 0

when

n = 1

. The full sampling procedure for

, starting

from white noise sample

, resembles Langevin dynamics

where we sample from the most noise-perturbed distribution

and reduce the magnitude of the noise scale until we reach

the smallest one.

of 11

25墨值下载

关注

评论