Learning A Sparse Transformer Network for Effective Image Deraining.pdf

dong

448

10页

0次

2023-10-31

25墨值下载

Learning A Sparse Transformer Network for Effective Image Deraining

Xiang Chen

Hao Li

Mingqiang Li

Jinshan Pan

School of Computer Science and Engineering, Nanjing University of Science and Technology

Information Science Academy, China Electronics Technology Group Corporation

Abstract

Transformers-based methods have achieved signiﬁcant

performance in image deraining as they can model the

non-local information which is vital for high-quality im-

age reconstruction. In this paper, we ﬁnd that most ex-

isting Transformers usually use all similarities of the to-

kens from the query-key pairs for the feature aggrega-

tion. However, if the tokens from the query are differ-

ent from those of the key, the self-attention values esti-

mated from these tokens also involve in feature aggregation,

which accordingly interferes with the clear image restora-

tion. To overcome this problem, we propose an effective

DeRaining network, Sparse Transformer (DRSformer) that

can adaptively keep the most useful self-attention values

for feature aggregation so that the aggregated features bet-

ter facilitate high-quality image reconstruction. Speciﬁ-

cally, we develop a learnable top-k selection operator to

adaptively retain the most crucial attention scores from the

keys for each query for better feature aggregation. Si-

multaneously, as the naive feed-forward network in Trans-

formers does not model the multi-scale information that is

important for latent clear image restoration, we develop

an effective mixed-scale feed-forward network to gener-

ate better features for image deraining. To learn an en-

riched set of hybrid features, which combines local con-

text from CNN operators, we equip our model with mix-

ture of experts feature compensator to present a coop-

eration reﬁnement deraining scheme. Extensive experi-

mental results on the commonly used benchmarks demon-

strate that the proposed method achieves favorable perfor-

mance against state-of-the-art approaches. The source code

and trained models are available at https://github.

com/cschenxiang/DRSformer.

1. Introduction

Single image deraining is a typical low-level vision prob-

lem emerging in the last decade. It aims to recover the clean

Corresponding author.

(a) Rainy Input (b) Uformer [48] (c) Restormer [58]

(d) IDT [50] (e) Ours (f) Ground Truth

Figure 1. Image deraining results between our method and recent

Transformer-based methods [48,50,58]. Our method can generate

high-quality image with more accurate detail and texture recovery.

image from the observed rainy one. As the clear image and

rain streaks are unknown, it is an ill-posed inverse problem.

To solve this problem, early approaches [20, 24, 60] usu-

ally impose various priors based on statistical properties of

rain streaks and clear images. In fact, these handcrafted pri-

ors are not robust to complex and varying rainy scenarios,

which limit the deraining performance.

Recently, numerous learning-based methods [4, 19, 23,

36, 52, 53, 56] have resorted to diverse CNN architectures

as a preferable choice compared to traditional algorithms.

However, the intrinsic characteristics of convolutional op-

eration, i.e, local receptive ﬁelds and independence of input

content, hinder the model’s capacity to eliminate long-range

rain degradation perturbation. To alleviate such limitations,

Transformers [2, 26, 35, 50] have been applied to image de-

raining and have achieved decent performance as they can

better model the non-local information for high-quality im-

age reconstruction. Nevertheless, the image details, which

are local features of images, are not modeled well by these

approaches when restoring clear images as shown in Fig-

ure 1. One main reason is that the self-attention in Trans-

formers does not model the local invariant properties that

arXiv:2303.11950v1 [cs.CV] 21 Mar 2023

CNNs do well. Since rain streaks tend to confuse with back-

ground details in local regions, recent studies [5, 18, 57] try

to mitigate such drawbacks by combining CNN operations

and Transformers for boosting image deraining, where the

Transformers based on the standard formulations.

We note that the standard Transformers [40] usually use

all attention relations based on the query-key pairs to ag-

gregate features. As the tokens from the key are not always

relevant to those from the query, using the self-attention val-

ues estimated from these tokens in the feature aggregation

interferes with the following latent clear image restoration.

The root cause behind this deﬁciency lies in that, the na-

tive dense calculation pattern of self-attention ampliﬁes rel-

atively smaller similarity weights, making feature interac-

tion and aggregation process susceptible to implicit noises.

This also naturally leads to corresponding redundant or ir-

relevant representations are still taken into consideration

when modeling global feature dependencies [44, 64]. Thus,

these ﬁndings motivate us to explore the most useful self-

attention values so that we can make full use of the features

for better image restoration.

To this end, we develop an effective sparse Transformer

network for image deraining, named as DRSformer. Specif-

ically, the key component of the proposed framework is

the sparse Transformer block (STB) which contains a top-

k sparse attention (TKSA) that keeps the most useful self-

attention values for feature aggregation and a mixed-scale

feed-forward network (MSFN) that explores the multi-scale

features for better image deraining. First, we design the top-

k attention mechanism to replace the vanilla self-attention

[40]. The TKSA keeps the largest K similarity scores be-

tween the queries and the keys for the self-attention comput-

ing, thereby facilitating better feature aggregation. Further-

more, the developed MSFN further explores the multi-scale

information to better improve the aggregated features. Fi-

nally, based on the observation that rain distribution reveals

the degradation location and degree, we also introduce mix-

ture of experts feature compensator (MEFC) to provide col-

laborative reﬁnement for STB. With the above-mentioned

designs, our proposed method offers three-fold advantages:

(1) it can enjoy natural robustness in terms of less sensi-

tivity to useless feature interference, (2) it can not only en-

rich the locality but also empower the capability of global

feature exploitation, and (3) it can co-explore data (embod-

ied in MEFC) and content (embodied in STB) sparsity for

achieving deraining performance gains.

The main contributions are summarized as follows:

• We propose a sparse Transformer architecture to help

generate high-quality deraining results with more ac-

curate detail and texture recovery.

• We develop a simple yet effective learnable top-k se-

lection operator to adaptively maintain the most useful

self-attention values for better feature aggregation.

• We design an effective feed-forward network based on

mixed-scale fusion strategy to explore multi-scale rep-

resentations for better facilitating image deraining.

• Extensive experimental results on various benchmarks

demonstrate that our method achieves favorable per-

formance against state-of-the-art (SOTA) approaches.

2. Related Work

Single image deraining. Since image deraining is an ill-

posed problem, traditional methods [12, 20, 24, 30, 60] usu-

ally develop kinds of image priors to provide additional

constraints. However, these handcrafted priors tend to rely

on empirical observations and thus are not able to model

the inherent properties of clear images. To overcome this

problem, numerous CNN-based frameworks [53] have been

developed to solve image deraining and achieved decent

restoration performance. To better represent the rain distri-

bution, several studies take rain characteristics such as rain

direction [27], density [61], veiling effect [15] into account,

and optimize the network structure via recursive computa-

tion [19, 23, 36] or transfer mechanism [16, 49, 54, 55]. Al-

though these methods achieve better performance than the

hand-crafted prior-based ones, they have difﬁculty captur-

ing the long-range dependencies due to the intrinsic limi-

tations of convolution. Different CNN-based deraining ap-

proaches, we utilize the Transformer as the network back-

bone to model non-local information for image deraining.

Vision Transformers. Motivated by the great success of

the Transformers [7] in natural language processing (NLP)

[40] and high-level vision tasks [1, 28], Transformers have

been applied to image restoration [2, 13, 48, 51, 58] and per-

form better than the previous CNN-based baselines as they

are able to model non-local information. For the ﬁeld of im-

age rain removal, Jiang et al. [18] design a dynamic asso-

ciated deraining network by incorporating self-attention in

Transformer with a background recovery network. More re-

cently, Xiao et al. [50] elaborately develop image deraining

Transformer (IDT) with window-based and spatial-based

dual Transformer to achieve excellent results. Note that,

most existing methods rely on the dense dot-product self-

attention as the heart of Transformers. However, one short-

coming of this computation manner is that redundant or ir-

relevant features with smaller weights may interfere with

the attention map, which makes the output features contain

potential noises. In this work, we propose sparse attention

in Transformer to relieve the negligence of the most relevant

information faced by vanilla self-attention.

Sparse representation. With inspirations drawn from neu-

ral activity in biological brains, sparsity of hidden represen-

tation in deep neural networks as a tantalizing “free lunch”

emerges for both vision and NLP tasks [44, 64]. Indeed, it

is widely proven that sparse representation also plays a crit-

ical role in handling low-level vision problems, such as im-

of 10

25墨值下载

关注

评论