LightPainter.pdf - 墨天轮文档

LightPainter.pdf

落枫0907

11页

0次

2023-10-17

50墨值下载

LightPainter: Interactive Portrait Relighting with Freehand Scribble

Yiqun Mei

He Zhang

Xuaner Zhang

Jianming Zhang

Zhixin Shu

Yilin Wang

Zijun Wei

Shi Yan

HyunJoon Jung

Vishal M. Patel

Johns Hopkins University

Adobe Inc.

Figure 1. LightPainter is an interactive lighting editing system that takes in an input image with freehand scribbles drawn on top and

renders the correspondingly relit portrait. It enables creative portrait lighting editing (left) and allows users to reproduce a target lighting

effect with ease (right).

Abstract

Recent portrait relighting methods have achieved realis-

tic results of portrait lighting effects given a desired light-

ing representation such as an environment map. However,

these methods are not intuitive for user interaction and

lack precise lighting control. We introduce LightPainter, a

scribble-based relighting system that allows users to inter-

actively manipulate portrait lighting effect with ease. This

is achieved by two conditional neural networks, a delighting

module that recovers geometry and albedo optionally con-

ditioned on skin tone, and a scribble-based module for re-

lighting. To train the relighting module, we propose a novel

scribble simulation procedure to mimic real user scribbles,

which allows our pipeline to be trained without any human

annotations. We demonstrate high-quality and ﬂexible por-

trait lighting editing capability with both quantitative and

qualitative experiments. User study comparisons with com-

mercial lighting editing tools also demonstrate consistent

user preference for our method.

1. Introduction

Lighting is a fundamental aspect of portrait photograph,

as lights shape the reality, and give the work depth, color-

fulness and excitement. Professional photographers [17,38]

spend hours designing lighting such that shadow and high-

light are distributed accurately on the subject to achieve the

desired photographic look. Getting the exact lighting setups

requires years of training, expensive equipment, environ-

ment setup, timing, and costly teamwork. Recently, portrait

relighting techniques [20, 22, 33, 43, 45, 48, 53, 61, 63] al-

low users to apply a different lighting condition to a portrait

photo. These methods require a given lighting condition:

some use an exemplar image [42, 43], which lacks precise

lighting control and requires exhaustive image search to ﬁnd

the speciﬁc style; some use a high dynamic range (HDR)

environment maps [33, 45, 48, 53] that is difﬁcult and unin-

tuitive to interpret or edit.

Hand-drawn sketches and scribbles have been shown to

be good for user interaction and thus are widely used in vari-

ous image editing applications [6, 9, 10, 30,32,57]. Inspired

arXiv:2303.12950v1 [cs.CV] 22 Mar 2023

by this, we propose LightPainter, a scribble-based inter-

active portrait relighting system. As shown in Figure 1,

LightPainter is an intuitive and ﬂexible lighting editing sys-

tem that only requires casual scribbles drawn on the input.

Unlike widely-used lighting representations such as envi-

ronment maps and spherical harmonics, it is non-trivial to

interpret free-hand scribbles as lighting effects for a number

of challenges.

The ﬁrst challenge is simulating scribbles to mimic real

free-hand input as it is impractical to collect a large num-

ber of human inputs. In addition, unlike other sketch-based

editing tasks [6,9,10,30,32,57] where sketches can be com-

puted from edges or orientation maps, there is no conven-

tional way to connect scribbles with lighting effects. To

address such challenge, we propose a scribble simulation

algorithm that can generate a diverse set of synthetic scrib-

bles that mimic real human inputs. For an interactive re-

lighting task, scribbles should be ﬂexible and expressive:

easy to draw and accurately reﬂecting the lighting effect,

such as changes in local shading and color. Compared to a

shading map, scribbles are often “incomplete”: users tend

to sparsely place the scribbles on a few key areas on the

face. Therefore, we propose to use a set of locally con-

nected “shading stripes” to describe local shading patterns,

including shape, intensity, and color, and use them to simu-

late scribbles. To this end, we simulate scribbles by starting

from a full shading map and applying a series of operations

to generate coarse and sparse shading stripes. We show that

training with our synthetic scribbles enables the system to

generalize well to real user scribbles from human inputs,

with which our model can generate high-quality results with

desirable lighting effects.

The second challenge is how to effectively use local and

noisy scribbles to robustly represent portrait lighting that

is often a global effect. LightPainter uses a carefully de-

signed network architecture and training strategy to han-

dle these discrepancies. Speciﬁcally, we introduce a two-

step relighting pipeline to process sparse scribbles. The

ﬁrst stage produces a plausible completion of the shading

map from the input scribbles and the geometry; the sec-

ond stage reﬁnes the shading and renders the appearance

with a learned albedo map. We propose a carefully de-

signed neural network with an augmented receptive ﬁeld.

Compared with commonly-used UNet for portrait relight-

ing [21, 31, 33, 48], our design can better handle the sparse

scribbles and achieve geometry-consistent relighting.

Last, there is one major challenge in portrait relight-

ing that originates from the ill-posed nature of the intrin-

sic decomposition problem. That is to decouple albedo and

shading from an image. It is also difﬁcult to address with

a learning framework due to the extreme scarcity of real-

istic labeled data and inﬁnite possible lighting conditions

for a scene. In the context of portrait relighting, it means

recovering the true skin tone of a portrait subject is very

challenging [12, 49]. Instead of trying to collect a balanced

large-scale light-stage [8] dataset to capture the continuous

and subtle variations in different skin tones, we propose an

alternative solution dubbed SkinFill. We draw inspiration

from the standard makeup routine and design SkinFill to al-

low users to specify skin tone in our relighting pipeline. We

use a tone map, a per-pixel skin tone representation, to con-

dition the albedo prediction to follow the exact skin tone as

desired. This also naturally enables additional user control

at inference time.

Similar to prior work [33, 45, 62], we train our system

with a light stage [8] dataset. With our novel designs, Light-

Painter is a user-friendly system that enables creative and

interactive portrait lighting editing. We demonstrate the

simple and intuitive workﬂow of LightPainter through a

thorough user study. We show it generates relit portraits

with superior photo-realism and higher ﬁdelity compared to

state-of-the-art methods. We summarize our contributions

as follows:

• We propose LightPainter, a novel scribble-based por-

trait relighting system that offers ﬂexible user control,

allowing users to easily design portrait lighting effects.

• We introduce a novel scribble simulation algorithm

that can automatically generate realistic scribbles for

training. Combining it with a carefully designed neural

relighting module, our system can robustly generalize

to real user input.

• We introduce SkinFill to allow users to specify skin

tone in the relighting pipeline, which allows data-

efﬁcient training and offers additional control to ad-

dress potential skin tone data bias.

2. Related Work

Portrait Relighting: The pioneering work of Debevec et

al. [8] presents an advanced illumination rig (i.e. the light

stage) to capture per-person reﬂectance ﬁeld, which is used

to render the subject under novel illuminations. Such tech-

nique has been used to create training data for a number of

single-image relighting methods [31,33,45,59,62,63]. Por-

trait relighting has also been formulated as style transfer.

Shih et al. [42] employ a multi-scale technique to transfer

the local statistics of an exemplar to the target image. Shu et

al. [43] formulate the lighting transfer as a geometry-aware

mass transport problem with 3D morphable face model.

Using quotient image to achieve relighting is introduced

in [34, 41], where they multiply the source image with a

ratio map to render novel illuminations. Intrinsic decom-

position based approaches [3, 21, 24, 27, 29, 31, 33, 40, 48]

factorize a source image into geometry, reﬂectance, and il-

lumination, and apply novel lighting by conditioning on a

of 11

50墨值下载

lightpainter

关注

评论