CAP-VSTNet.pdf - 墨天轮文档

CAP-VSTNet.pdf

dong

19页

0次

2023-10-12

50墨值下载

CAP-VSTNet: Content Afﬁnity Preserved Versatile Style Transfer

Linfeng Wen

Chengying Gao

1 *

Changqing Zou

2,3

Sun Yat-Sen University

State Key Lab of CAD&CG, Zhejiang University

Zhejiang Lab

wenlf5@mail2.sysu.edu.cn mcsgcy@mail.sysu.edu.cn aaronzou1125@gmail.com

Figure 1. Our model produces more consistent stylization results with content afﬁnity preserved for photorealistic image style transfer

(left), and photorealistic video and artistic video style transfer (right). Animations can be found in the supplementary material.

Abstract

Content afﬁnity loss including feature and pixel afﬁnity

is a main problem which leads to artifacts in photorealis-

tic and video style transfer. This paper proposes a new

framework named CAP-VSTNet, which consists of a new re-

versible residual network and an unbiased linear transform

module, for versatile style transfer. This reversible resid-

ual network can not only preserve content afﬁnity but not

introduce redundant information as traditional reversible

networks, and hence facilitate better stylization. Empow-

ered by Matting Laplacian training loss which can address

the pixel afﬁnity loss problem led by the linear transform,

the proposed framework is applicable and effective on ver-

satile style transfer. Extensive experiments show that CAP-

VSTNet can produce better qualitative and quantitative re-

sults in comparison with the state-of-the-art methods.

1. Introduction

Photorealistic style transfer aims to reproduce content

image with the style from a reference image in a photore-

*Corresponding Author

alistic way. To be photorealism, the stylized image should

preserve clear content detail and consistent stylization of

the same semantic regions. Content afﬁnity preservation,

including feature and pixel afﬁnity preservation [23,25,28],

is the key to achieve both clear content detail and consistent

stylization in the transfer.

The framework of a deep learning based photorealis-

tic style transfer generally uses such an architecture: an

encoder module extracting content and style features, fol-

lowed by a transformation module to adjust features statis-

tics, and ﬁnally a decoder module to invert stylized feature

back to stylized image. Existing photorealistic methods typ-

ically employ pre-trained VGG [30] as encoder. Since the

encoder is speciﬁcally designed to capture object-level in-

formation for the classiﬁcation task, it inevitably results in

content afﬁnity loss. To reduce the artifacts, existing meth-

ods either use skip connection modules [2, 14, 40] or build

a shallower network [8, 23, 39]. However, these strategies,

limited by the image recovery bias, cannot achieve a perfect

content afﬁnity preservation on unseen images.

In this work, rather than use the traditional encoder-

transformation-decoder architecture, we resort to a re-

versible framework [1] based solution called CAP-VSTNet,

which consists of a speciﬁcally designed reversible residual

arXiv:2303.17867v1 [cs.CV] 31 Mar 2023

network followed by an unbiased linear transform module

based on Cholesky decomposition [19] that performs style

transfer in the feature space. The reversible network takes

the advantages of the bijective transformation and can avoid

content afﬁnity information loss during forward and back-

ward inference. However, directly using the reversible net-

work cannot work well on our problem. This is because re-

dundant information will accumulate greatly when the net-

work channel increases. It will further lead to content afﬁn-

ity loss and noticeable artifacts as the transform module is

sensitive to the redundant information. Inspired by knowl-

edge distillation methods [8, 35], we improve the reversible

network and employ a channel reﬁnement module to avoid

the redundant information accumulation. We achieve this

by spreading the channel information into a patch of the

spatial dimension. In addition, we also introduce cycle con-

sistency loss in CAP-VSTNet to make the reversible net-

work robust to small perturbations caused by numerical er-

ror.

Although the unbiased linear transform based on

Cholesky decomposition [19] can preserve feature afﬁnity,

it cannot guarantee pixel afﬁnity. Inspired by [25, 28], we

introduce Matting Laplacian [22] loss to train the network

and preserve pixel afﬁnity. Matting Laplacian [22] may

result in blurry images when it is used with another net-

work like one with an encoder-decoder architecture. But it

does not have this issue in CAP-VSTNet, since the bijective

transformation of reversible network theoretically requires

all information to be preserved.

CAP-VSTNet can be ﬂexibly applied to versatile style

transfer, including photorealistic and artistic image/video

style transfer. We conduct extensive experiments to eval-

uate its performance. The results show it can produce better

qualitative and quantitative results in comparison with the

state-of-the-art image style transfer methods. We show that

with minor loss function modiﬁcations, CAP-VSTNet can

perform stable video style transfer and outperforms existing

methods.

2. Related Work

2.1. Style Transfer

Gatys et al. [11] expose the powerful representation abil-

ity of deep neural networks and propose neural style transfer

by matching the correlations of deep features. Feed-forward

frameworks [17, 33, 38] are proposed to address the issue

of computational cost. To achieve universal style transfer,

transformation modules are proposed to adjust statistics of

deep features, such as the mean and variance [15] and the

inter-channel correlation [24].

Photorealistic style transfer requires that stylized image

should be undistorted and consistently stylized. DPST [28]

optimizes stylized image with regularization term computed

on Matting Laplacian [22] to suppress distortion. Pho-

toWCT [25] proposes post-processing algorithm by using

Matting Laplacian as afﬁnity matrix to reduce artifacts.

However, both of these methods may blur the stylized im-

ages instead of preserving the pixel afﬁnity. The follow-

ing works [2,8,40] mainly focus on preserving clear details

and speeding up processing by designing skip connection

module or shallower network. Content afﬁnity preservation

including feature and pixel afﬁnity preservation remains an

unsolved challenge.

Recently, versatile style transfer has received a lot of

attention. Many approaches focus on exploring a gen-

eral framework capable of performing artistic, photoreal-

istic and video style transfer. Li et al. [23] propose a lin-

ear style transfer network and a spatial propagation net-

work [27] for artistic and photorealistic style transfer, re-

spectively. DSTN [14] introduces a uniﬁed architecture

with domain-aware indicator to adaptively balance between

artistic and photorealistic stylization. Chiu et al. [7] pro-

pose an optimization-based method to achieve fast artistic

or photorealistic style transfer by simply adjusting the num-

ber of iterations. Chen et al. [6] extend contrastive learn-

ing to artistic image and video style transfer by considering

internal-external statistics. Wu et al. [39] apply contrastive

learning by incorporating neighbor-regulating scheme to

preserve the coherence of the content source for artistic and

photorealistic video style transfer. While achieving versa-

tile style transfer, VGG-based networks suffer from incon-

sistent stylization due to content afﬁnity loss. We show that

preserving content afﬁnity can improve image consistent

stylization and video temporal consistency.

2.2. Reversible Network

Dinh et al. [9] ﬁrst propose an estimator that learns a bi-

jective transform between data and latent space, which can

be seen as a perfect auto-encoder pair as it naturally satis-

ﬁes reconstruction term of auto-encoder [4, 34]. Follow-up

work by Dinh et al. [10] introduces new transformation that

breaks the unit determinant of Jacobian to address volume-

preserving mapping. Glow [20] proposes a simple type

of generative ﬂow building on the works by Dinh [9, 10].

Since each layer’s activation of reversible network can be

exactly reconstructed from the next layer’s, RevNet [12]

and Reformer [21] present reversible residual layers to ad-

dress memory consumption during deep network training.

i-RevNet [16] builds an invertible type of RevNet with in-

vertible down-sampling module. i-ResNet [3] inverts resid-

ual mapping by using Banach ﬁxed point theorem to address

the restriction of reversible network architecture.

Recently, An et al. [1] apply ﬂow-based model [20] to

address the content leak problem for artistic style transfer.

However, content afﬁnity may not be preserved due to trans-

formation module and redundant information, which leads

of 19

50墨值下载

cap-vstnet

关注

评论