NeuFace Realistic 3D Neural Face Rendering from Multi-view Images.pdf

smith0907

136

20页

0次

2023-09-16

100墨值下载

NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images

Mingwu Zheng

1,2

, Haiyu Zhang

1,2

, Hongyu Yang

, Di Huang

1,2,4

State Key Laboratory of Software Development Environment, Beihang University, Beijing, China

School of Computer Science and Engineering, Beihang University, Beijing, China

Institute of Artiﬁcial Intelligence, Beihang University, Beijing, China

Hangzhou Innovation Institute, Beihang University, Hangzhou, China

{zhengmingwu, zhyzhy, hongyuyang, dhuang}@buaa.edu.cn

Abstract

Realistic face rendering from multi-view images is bene-

ﬁcial to various computer vision and graphics applications.

Due to the complex spatially-varying reﬂectance proper-

ties and geometry characteristics of faces, however, it re-

mains challenging to recover 3D facial representations both

faithfully and efﬁciently in the current studies. This paper

presents a novel 3D face rendering model, namely Neu-

Face, to learn accurate and physically-meaningful under-

lying 3D representations by neural rendering techniques.

It naturally incorporates the neural BRDFs into physically

based rendering, capturing sophisticated facial geometry

and appearance clues in a collaborative manner. Speciﬁ-

cally, we introduce an approximated BRDF integration and

a simple yet new low-rank prior, which effectively lower the

ambiguities and boost the performance of the facial BRDFs.

Extensive experiments demonstrate the superiority of Neu-

Face in human face rendering, along with a decent gener-

alization ability to common objects. Code is released at

NeuFace.

1. Introduction

Rendering realistic human faces with controllable view-

points and lighting is now becoming ever increasingly im-

portant with its applications ranging from game production,

movie industry, to immersive experiences in the Metaverse.

Various factors, including the sophisticated geometrical dif-

ferences among individuals, the person-speciﬁc appearance

idiosyncrasies, along with the spatially-varying reﬂectance

properties of skins, collectively make faithful face rendering

a rather difﬁcult problem.

According to photogrammetry, the pioneering studies on

this issue generally leverage complex active lighting setups,

e.g., LightStage [9], to build 3D face models from multiple

Corresponding author.

Rendered

Diffuse

Specular

Geometry

Relighting

Unprocessed Facial Multiview Images

…

Figure 1. Demonstration of the face rendering results and recov-

ered underlying 3D representations.

photos of an individual, where accurate shape attributes and

high-quality diffuse and specular reﬂectance properties are

commonly acknowledged as the premises of its success. An

elaborately designed workﬂow is required, typically involv-

ing a series of stages such as camera calibration, dynamic

data acquisition, multi-view stereo, material estimation, and

texture parameterization [44]. While a compelling and con-

vincing 3D face model can be ﬁnally obtained, this output

highly depends on the expertise of the engineers and artists

with signiﬁcant manual efforts, as the multi-step process in-

evitably brings diverse optimization goals.

Recently, 3D neural rendering, which offers an end-to-

end alternative, has demonstrated promising performance in

recovering scene properties from real-world imageries, such

as view-dependent radiance [27, 29, 37, 39, 49] and geome-

try [34, 50, 51, 56, 57]. It is mainly credited to the disentan-

glement of the learnable 3D representations and the differ-

entiable image formation process, free of the tedious pho-

togrammetry pipeline. However, like classical function ﬁt-

ting, inverse rendering is fundamentally under-constrained,

which may incur badly-conditioned ﬁts of the underly-

ing 3D representations, especially for intricate cases, e.g.,

non-Lambertian surfaces with view-dependent highlights.

With the trend in the combination of computer graphics

arXiv:2303.14092v2 [cs.CV] 27 Mar 2023

and learning techniques, several attempts take advantage

of physically motivated inductive biases and present Phys-

ically Based Rendering (PBR) [15, 32, 49, 58], where Bidi-

rectional Reﬂectance Distribution Functions (BRDFs) are

widely adopted. By explicitly mimicking the interaction

of the environment light with the scene, they facilitate net-

work optimization and deliver substantial gains. Unfortu-

nately, the exploited physical priors are either heuristic or

analytic [8, 21, 46], limited to a small set of real-world ma-

terials, e.g., metal, incapable of describing human faces.

For realistic face rendering, the most fundamental issue

lies in accurately modeling the optical properties of multi-

layered facial skin [22]. In particular, the unevenly dis-

tributed ﬁne-scale oily layers and epidermis reﬂect the inci-

dent lights irregularly, leading to complex view-dependent

and spatially-varying highlights. This characteristic and the

low-textured nature of facial surfaces strongly amplify the

shape-appearance ambiguity. Moreover, subsurface scatter-

ing between the underlying dermis and other skin layers fur-

ther complicates this problem.

In this paper, we follow the PBR paradigm for its poten-

tial in learning 3D representations and make the ﬁrst step

towards realistic 3D neural face rendering, mainly target-

ing complex skin reﬂection modeling. Our method, namely

NeuFace, is able to recover faithful facial reﬂectance and

geometry from only multi-view images. Concretely, we es-

tablish a PBR framework to learn neural BRDFs to describe

facial skin, which simulates physically-correct light trans-

port with a much higher representation capability. By using

a differentiable Signed Distance Function (SDF) based rep-

resentation, i.e. ImFace [63], as the shape prior, the facial

appearance and geometry ﬁeld can be synchronously opti-

mized in inverse rendering.

Compared to the analytic BRDFs, the neural ones allow

richer representations for sophisticated material like facial

skin. In spite of this superiority, such representations pose

challenges to computational cost and data demand during

training. To tackle these difﬁculties, the techniques in real-

time rendering [1] are adapted to separate the hemisphere

integral of neural BRDFs, where the material and light in-

tegrals are individually learned instead, bypassing the mas-

sive Monte-Carlo sampling phase [35] required by numer-

ical solutions. Furthermore, a low-rank prior is introduced

into the spatially-varying facial BRDFs, which greatly re-

stricts the solution space thereby diminishing the need for

large-scale training observations. These model designs in-

deed enable NeuFace to accurately and stably describe how

the light interacts with the facial surface as in the real 3D

space. Fig. 1 displays an example.

The main contributions of this study include: 1) A novel

framework with naturally-bonded PBR as well as neural

BRDF representations, which collaboratively captures fa-

cial geometry and appearance properties in complicated fa-

cial skin. 2) A new and simple low-rank prior, which sig-

niﬁcantly facilitates the learning of neural BRDFs and im-

proves the appearance recovering performance. 3) Impres-

sive face rendering results from only multi-view images,

applicable to various applications such as relighting, along

with a decent generalization ability to common objects.

2. Related Work

We restrict the discussion speciﬁcally to static facial ge-

ometry and appearance capturing and 3D neural rendering.

Please refer to [10, 22, 45] for more in-depth discussion.

Face Capturing. Its goal is to render a realistic 3D face

under arbitrary lighting condition. Existing methods gener-

ally take advantage of photogrammetry techniques to esti-

mate facial geometries and appearances, requiring massive

manual efforts. In this case, they typically decompose the

problem where facial geometry is pre-captured by an intri-

cate multi-view stereo process [3, 4]. However, facial ap-

pearances with reﬂectances are still hard to acquire due to

complex interactions between light and skin. The initial at-

tempts [9, 17, 53] tackle the challenge by densely capturing

the per-pixel facial reﬂectance at the cost of extensive data

acquisition and specialized equipment. Subsequently, gra-

dient [12,20, 26] or polarized illumination [12, 13, 26, 40] is

explored to reduce the memory cost, where most efforts are

paid to the well-conditioned ﬁtting of BRDFs. In contrast to

the studies above, our solution is truly end-to-end and only

observes facial skin under a single, unknown illumination

condition, without cumbersome capturing settings.

3D Neural Rendering. The recent advance in this ﬁeld,

like NeRF [30], have already revolutionized the paradigm

of multi-view reconstruction. With learnable volumetric

representations (e.g., neural ﬁeld [30, 57], grid [11], and

hybrid [31]) and analytic differentiable forward mapping,

scene properties can be directly inferred from only 2D

imageries. While reporting impressive results on novel

view synthesis [27, 29, 37, 39, 49] or geometry reconstruc-

tion [34, 50, 51, 56, 57], studies still suffer from 2D-3D am-

biguities, leading to that realistic appearances and accurate

geometries can hardly be established simultaneously [59].

Such an issue is more prominent for facial skin due to the

rather complicated reﬂection properties. Ref-NeRF [49]

makes a step forward to the goal of both accurate surface

normals and glossy appearances, which is achieved by repa-

rameterizing the radiance in classical NeRF with reﬂec-

tion direction. It validates the signiﬁcance of physical laws

in disambiguity. [15, 23, 32, 43, 58] go further with PBR

pipelines, delivering improved quality and supporting re-

lighting simultaneously. Nevertheless, simpliﬁed material

models are utilized or assumed, which are incapable of han-

dling complex materials like skin. [25, 61, 62] train a neu-

ral material model from measurement data. But acquiring

such data is impractical for live biotissue. By contrast, our

of 20

100墨值下载

关注

评论