04-Adaptive Data Augmentation for Supervised Learning over Missing Data.pdf

Express

124

13页

0次

2021-11-09

100墨值下载

Adaptive Data Augmentation for Supervise d Learning over

Missing Data

Tongyu Liu

Renmin University of China

ltyzzz@ruc.edu.cn

Ju Fan

Renmin University of China

fanj@ruc.edu.cn

Yinqing Luo

Renmin University of China

lyqroy@ruc.edu.cn

Nan Tang

QCRI, HBKU

ntang@hbku.edu.qa

Guoliang Li

Tsinghua University

liguoliang@tsinghua.edu.cn

Xiaoyong Du

Renmin University of China

duyong@ruc.edu.cn

ABSTRACT

Real-world data is dirty, which causes serious problems in (super-

vised) machine learning (ML). The widely used practice in such

scenario is to rst repair the labeled source (a.k.a. train) data using

rule-, statistical- or ML-based methods and then use the “repaired”

source to train an ML model. During production, unlabeled target

(a.k.a. test) data will also be repaired, and is then fed in the trained

ML model for prediction. However, this process often causes a per-

formance degradation when the source and target datasets are dirty

with dierent noise patterns, which is common in practice.

In this paper, we propose an adaptive data augmentation ap-

proach, for handling missing data in supervised ML. The approach

extracts noise patterns from target data, and adapts the source data

with the extracted target noise patterns while still preserving su-

pervision signals in the source. Then, it patches the ML model by

retraining it on the adapted data, in order to better serve the tar-

get. To eectively support adaptive data augmentation, we propose

a novel generative adversarial network (GAN) based framework,

called DAGAN, which works in an unsupervised fashion. DAGAN

consists of two connected GAN networks. The rst GAN learns the

noise pattern from the target, for target mask generation. The sec-

ond GAN uses the learned target mask to augment the source data,

for source data adaptation. The augmented source data is used to

retrain the ML model. Extensive experiments show that our method

signicantly improves the ML model performance and is more ro-

bust than the state-of-the-art missing data imputation solutions for

handling datasets with dierent missing value patterns.

PVLDB Reference Format:

Tongyu Liu, Ju Fan, Yinqing Luo, Nan Tang, Guoliang Li, and Xiaoyong Du.

Adaptive Data Augmentation for Supervised Learning over Missing Data.

PVLDB, 14(7): XXX-XXX, 2021.

doi:10.14778/3450980.3450989

PVLDB Artifact Availability:

The source code, data, and/or other artifacts have been made available at

https://github.com/ruc-datalab/dagan.

This work is licensed under the Creative Commons BY-NC-ND 4.0 International

License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of

this license. For any use beyond those covered by this license, obtain permission by

emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights

licensed to the VLDB Endowment.

Proceedings of the VLDB Endowment, Vol. 14, No. 7 ISSN 2150-8097.

doi:10.14778/3450980.3450989

∗

Ju Fan is the corresponding author.

1 INTRODUCTION

Machine learning (ML) techniques have been deployed in (almost)

all applications nowadays. Among all the techniques, supervised

ML, such as classication and regression, is the most prevalent [

Typically, a supervised ML application follows two basic steps. First,

data scientists or ML engineers prepare a labeled source dataset

(a.k.a. training dataset) and train a supervised learning model, e.g.,

a classier, on the dataset. Then, the trained model will be deployed

in one or many production environments to make prediction on

the target datasets (a.k.a. test datasets) [27].

In practice, unfortunately, the ML models are dicult to maintain

in the production environments. One central challenge are unex-

pected errors in the target data, which is fed into the ML models

at prediction time [

]. Among all the data errors [

], missing

data [

] is a serious problem that data scientists need to account

for everyday. There are a multitude of reasons why they occur:

ranging from human errors during data entry, incorrect sensor

readings, to software bugs in data science pipelines [

]. Moreover,

dierent from other types of errors (e.g., wrong names/addresses,

unnormalized values, and violations of integrity constraints) that

sometimes can be remained as they are, for ML modeling, these

missing data elds must be deleted or imputed rst.

Consider an exemplary scenario as shown in Figure 1. Suppose

that a hospital trains a classier that predicts cardiovascular disease

(i.e.,

cardio

) for patients based on a labeled source dataset

which contains examination features, such as cholesterol (

chol

)

and glucose (

gluc

), patient-reported features, such as smoking

(

smoke

) and alcohol intake (

alcohol

), and demographics of patients,

such as

age

. We can observe that

contains missing values in

attributes

smoke

and

alcohol

, possibly because some patients may

not want to report their habits. However, when being deployed

in a production environment for prediction, the missing pattern

of the unlabeled target data

might be dierent, as shown in

Figure 1(b). There could be many reasons for such noise shift. For

example, the model is deployed to predict another patient cohort

or even in another hospital, where patients have missing values

in examination features instead of smoking or alcohol habits. Not

surprisingly, the model performance often degrades signicantly

when encountering the noise shift in the target data.

Limitations of Missing Data Imputation Methods.

To tackle

the problem, the existing works have proposed many methods for

imputing missing values [

], using mean imputation, regression

imputation, maximum likelihood, multiple imputation, etc. Ideally,

age chol gluc smoke alcohol cardio

25 1 NA 0 1 ?

37 NA 3 0 0 ?

40 3 NA 1 1 ?

72 3 2 0 0 ?

age chol gluc smoke alcohol cardio

30 2 3 0 0 no

35 2 1 1 0 no

50 3 3 NA 1 yes

65 2 3 1 NA yes

70 3 1 1 0 no

(a) A labeled source data ( )

(b) An unlabeled target data ( )

labels

no labels

Figure 1: Source and target with dierent missing patterns.

if both

and

can be imputed to be their ground-truth, we

can solve the problem. In practice, unfortunately, this is extremely

hard, because of the high cost of data cleaning [

]. Thus, in most

cases, practitioners will pick data imputation methods from the

decades-long eort on this eld [

]. However, regardless of how so-

phisticated data cleaning techniques the practitioners may choose,

these techniques may have suboptimal results, i.e., the repaired

data is not the ground-truth. Subsequently, data imputation sep-

arately for

and

might cause an even bigger divergence on

data distributions, thus degrading the model performance. Consider

our previous example shown in Figure 1: an imputation method

may lead to a result that distribution of attribute

gluc

in target

diverges from that in source

. This would result in a well-

recognized problem in ML, called dataset shift [

], which

may severely aect the performance of ML models.

Our Solution.

Because nding the ground-truth for

and

is practically infeasible, one alternative is to extract some “signal”

(or noise pattern) from the target, which can be used to adapt the

source and retrain the model, in order to better serve the target.

Based on this idea, we introduce a novel approach, dubbed adap-

tive data augmentation. It rst adapts source data

learning missing patterns from an unlabeled dataset in the target

and then “patches” the ML model by retraining the model on

The objective of the adaptation is two-fold. First, the adapted

dataset

should preserve the supervision signals in

, e.g., the

conditional feature distribution with respect to label remains con-

stant. For example, we do not want

to contain tuples that violate

the potential correlation between features and the

cardio

outcome.

Second, the adapted dataset

should be similar to our target

dataset

. As we consider the case that source and target have the

same data distribution but dierent missing patterns, this essen-

tially means that the missing pattern between

and

should

be as similar as possible.

The main challenges are to learn the (missing) noise pattern from

the target and adapt it to the source. We propose a novel generative

adversarial network (GAN) [

] based framework, namely DAGAN.

DAGAN is an end-to-end learning approach that consists of two

connected GAN networks. (i) Target mask generation: The rst GAN

is to learn the noise pattern from the target, e.g., younger people

(i.e., age

≤

40) tend to have more incomplete examination results in

attributes

chol

and

gluc

as shown in Figure 1. (ii) Source data adap-

tation: The second GAN uses the learned target mask to “translate”

the source data to an adapted data, which is indistinguishable from

the real target data. Another challenge is to apply the adapted data

to improve the ML model. We retrain the model using the adapted

data and get a “patched” model, to better serve the target data

We also study a more practical scenario with multiple dirty target

datasets

, . . . , D

}

, each with a dierent noise pattern. We

propose a method to retrain one single patched model for all targets,

by using Group Distributionally Robust Optimization (GDRO) [

Contributions. We make the following notable contributions.

(1) We propose a novel framework to adaptively augment the source

data to be alike the target data and retrain an ML model using the

augmented source data, which can better serve the prediction on

the unseen target data (Sections 2 and 3).

(2) We propose a novel GAN-based approach DAGAN for adaptive

data augmentation (Section 4) and introduce eective model re-

training methods for patching the supervised ML model (Section 5).

(3) We experiment on real-world datasets and show that our adap-

tive data augmentation using DAGAN can improve the model per-

formance, and is robust for dierent missing data patterns, including

missing not at random (MNAR), missing at random (MAR), and

missing completely at random (MCAR) (Section 6).

2 PRELIMINARIES

Data Model.

We consider a relational table

with attributes

A = {A

, A

, . . . , A

}

and with tuples

(1)

, x

(2)

, . . . , x

(M )

}

. In

particular, we consider both categorical (nominal) and numerical

(either discrete or continuous) attributes. We use

to denote the

value of attribute A

in tuple x (i.e., x

= x[A

]).

Missing Data.

We consider that

contains missing values: each

tuple

may have some attributes that are not observed. To formalize

these missing values, like the existing works [

], we introduce

a mask vector

m ∈ {

}

to indicate which attributes in

are

observed (i.e., not missing):

∈ x

is observed if

1 (

is the

-th entry of

), otherwise

is missing and is denoted as

for

ease of presentation. Thus, our dataset

can be represented as

D = {(x

(i)

, m

(i)

)}. For a clearer notation, we use x

to denote the

complete (i.e., ground-truth) tuple without missing values, and

to denoted the observed incomplete tuple, and we have:

x = ψ (x

, m) = x

⊙ m + NA · (1 − m), (1)

where

⊙

is element-wise multiplication and

(

− m)

is the comple-

ment of

. We consider three common missing mechanisms [

]

to model the conditional distribution

p(m|x

)

of mask given the

ground-truth data. (1) Missing completely at random (MCAR) : mask

is independent with the data, i.e.,

p(m|x

) = p(m )

; (2) Missing at

random (MAR): mask

is only dependent on the observed data, i.e.,

p(m|x

) = p(m|x)

; (3) Missing not at random (MNAR): mask

dependent on both observed and unobserved data.

Example 1. Consider Figure 1. We can observe that some tuples

contain missing values. For example, tuple

is incomplete as value

smoke

is missing. Thus,

corresponds to a mask vector

m =

[1, 1, 1, 0, 1]

. Suppose that the ground-truth of

x = (

)

We represent the observed tuple corresponding to

obs

= x ⊙

m + NA · (1 − m) = (50, 3, 3, NA, 1).

of 13

100墨值下载

paper 李国良

关注

评论