深层语义的脸部去模糊.pdf

poPoq

11页

0次

2021-05-01

50墨值下载

Deep Semantic Face Deblurring

Ziyi Shen

Wei-Sheng Lai

Tingfa Xu

1∗

Jan Kautz

Ming-Hsuan Yang

2,4

Beijing Institute of Technology

University of California, Merced

Nvidia

Google Cloud

https://sites.google.com/site/ziyishenmi/cvpr18_face_deblur

Abstract

In this paper, we present an effective and efﬁcient face

deblurring algorithm by exploiting semantic cues via deep

convolutional neural networks (CNNs). As face images are

highly structured and share several key semantic compo-

nents (e.g., eyes and mouths), the semantic information of

a face provides a strong prior for restoration. As such, we

propose to incorporate global semantic priors as input and

impose local structure losses to regularize the output within

a multi-scale deep CNN. We train the network with percep-

tual and adversarial losses to generate photo-realistic re-

sults and develop an incremental training strategy to handle

random blur kernels in the wild. Quantitative and qualita-

tive evaluations demonstrate that the proposed face deblur-

ring algorithm restores sharp images with more facial de-

tails and performs favorably against state-of-the-art meth-

ods in terms of restoration quality, face recognition and ex-

ecution speed.

1. Introduction

Single image deblurring aims to recover a clear image

from a single blurred input image. Conventional methods

model the blur process (assuming spatially invariant blur) as

the convolution operation between a latent clear image and

a blur kernel, and formulate this problem based on the max-

imum a posteriori (MAP) framework. As the problem is ill-

posed, the state-of-the-art algorithms rely on natural image

priors (e.g., L

gradient [48] and dark channel prior [31]) to

constrain the solution space.

While existing image priors are effective for deblurring

natural images, the underlying assumption may not hold

for images from speciﬁc categories, e.g., text, face and

low-light conditions. Therefore, numerous approaches ex-

ploit domain-speciﬁc priors or strategies, such as L

in-

tensity [30] for text images and light streaks [13] for ex-

tremely low-light images. As face images typically have

fewer textures and edges for estimating blur kernels, Pan et

∗

Corresponding author

(a) (b) (c) (d)

Figure 1. Face deblurring results. We exploit the semantic in-

formation of face within an end-to-end deep CNN for face image

deblurring. (a) Ground truth images (b) Blurred images (c) Ours

w/o semantics (d) Ours w/ semantics.

al. [29] propose to search a similar face exemplar from an

external dataset and extract the contour as reference edges.

However, a similar reference image may not always exist to

cover the diversity of face images in the wild. Furthermore,

those methods based on the MAP framework typically en-

tail heavy computational cost due to the iterative optimiza-

tion of latent images and blur kernels. The long execution

time limits the applications on resource-sensitive platforms,

e.g., cloud and mobile devices.

In this work, we focus on deblurring face images and

propose an efﬁcient as well as effective solution using

deep CNNs. Since face images are highly structured and

composed of similar components, the semantic information

serves as a strong prior for restoration. Therefore, we pro-

pose to leverage the face semantic labels as global priors

and local constraints for deblurring face images. Speciﬁ-

cally, we ﬁrst generate the semantic labels of blurred input

images using a face parsing network. The face deblurring

network then takes the blurred image and semantic labels

as input to restore a clear image in a coarse-to-ﬁne man-

arXiv:1803.03345v2 [cs.CV] 16 Mar 2018

ner. To encourage the network for generating ﬁne details,

we further impose a local structure loss on important face

components (e.g., eyes, noses, and mouths). Figure 1 shows

deblurred examples with and without the proposed seman-

tic priors and losses. The proposed method is able to recon-

struct better facial details than the network trained with only

the pixel-wise L

loss function (i.e., without using seman-

tics). As our method is end-to-end without any blur kernel

estimation or post-processing, the execution time is much

shorter than the state-of-the-art MAP-based approaches.

To handle blurred images produced by unknown blur

kernels, existing methods typically synthesize blur kernels

by modeling the camera trajectories [4, 12] and generate a

large number of blurred images for training. Instead of si-

multaneously using all synthetic blurred images for train-

ing, we propose an incremental training strategy by ﬁrst

training the network on a set of small blur kernels and then

incorporating larger blur kernels sequentially. We show that

the proposed incremental training strategy facilitates the

convergence and improves the performance of our deblur-

ring network on various sizes of blur kernels. Finally, we

impose a perceptual loss [14] and an adversarial loss [10] to

generate photo-realistic deblurred results.

We make the following contributions in this work:

• We propose a deep multi-scale CNN that exploits

global semantic priors and local structural constraints

for face image deblurring.

• We present an incremental strategy to train CNNs to

better handle unknown motion blur kernels.

• We demonstrate that the proposed method performs fa-

vorably against state-of-the-art deblurring approaches

in terms of restoration quality, face recognition and ex-

ecution speed.

2. Related Work

Single image deblurring can be categorized into non-

blind and blind deblurring based on whether the blur kernel

is available or not. We focus our discussion on blind image

deblurring in this section.

Generic methods. The recent progress in single im-

age blind deblurring can be attributed to the development

of effective natural image priors, including sparse image

gradient prior [8, 23], normalized sparsity measure [17],

patch prior [42], L

gradient [48], color-line model [18],

low-rank prior [34], self-similarity [27] and dark channel

prior [31]. Through optimizing the image priors within the

MAP framework, those approaches implicitly restore strong

edges for estimating the blur kernels and latent sharp im-

ages. However, solving complex non-linear priors involve

several optimization steps and entail high computational

loads. As such, edge-selection based methods [6, 46] adopt

simple image priors (e.g., L

gradients) with image ﬁlters

(e.g., shock ﬁlter) to explicitly restore or select strong edges.

While generic image deblurring methods demonstrate state-

of-the-art performance, face images have different statisti-

cal properties than natural scenes and cannot be restored

well using the above approaches.

Domain-speciﬁc methods. To handle images from spe-

ciﬁc categories, several domain-speciﬁc image deblurring

approaches have been developed. Pan et al. [30] introduce

the L

-regularized priors on both intensity and image gradi-

ents for text image deblurring as text images usually contain

nearly uniform intensity. To handle extreme cases such as

low-light images, Hu et al. [13] detect the light streaks in

images for estimating blur kernels. Anwar et al. [2] pro-

pose a frequency-domain class-speciﬁc prior to restore the

band-pass frequency components. In addition, a number

of approaches use reference images as guidance for non-

blind [43] and blind deblurring [11]. However, the perfor-

mance of such methods hinges on the similarity of the ref-

erence images and quality of dense correspondence.

As face images have fewer textures and edges, existing

algorithms based on implicit or explicit edge restoration are

less effective. Pan et al. [29] search for similar faces from

a face dataset and extract reference exemplar contours for

estimating blur kernels. However, this approach requires

manual annotations of the face contours and involves com-

putationally expensive optimization processes of blur ker-

nels and latent images in the MAP framework. In contrast,

we train an end-to-end deep CNN to bypass the blur ker-

nel estimation step and do not use any reference images or

manual annotations when deblurring an image.

CNN-based methods. Deep CNNs have been adopted

for several image restoration tasks, such as denois-

ing [26], JPEG deblocking [7], dehazing [35] and super-

resolution [16, 19]. Recent approaches apply deep CNNs

for image deblurring in several aspects, including non-blind

deconvolution [37, 47, 50], blur kernel estimation [38] and

dynamic scene deblurring [28]. Chakrabarti et al. [4] train

a deep network to predict the Fourier coefﬁcients of a de-

convolution ﬁlter. Despite computational efﬁciency, these

CNN-based methods do not perform as well as the state-of-

the-art MAP-based approaches, especially on large motion

kernels.

Since text images usually contain uniform intensities

with fewer texture regions, an end-to-end deep network [12]

performs well, especially under large noise levels. Xu

et al. [49] aim to jointly deblur and super-resolve low-

resolution blurred face and text images, which are typically

degraded by Gaussian-like blur kernels. In this work, we

focus on deblurring face images from complex motion blur.

We exploit global and local semantic cues as well as per-

ceptual [14] and adversarial [10] losses to restore photo-

realistic face images with ﬁne details.

of 11

50墨值下载

自动驾驶

关注

评论