视觉问答的视觉-文本注意力焦点.pdf

poPoq

125

10页

0次

2021-05-02

50墨值下载

Focal Visual-Text Attention for Visual Question Answering

Junwei Liang

Lu Jiang

Liangliang Cao

Li-Jia Li

Alexander Hauptmann

Carnegie Mellon University

Google Inc.

HelloVera AI

{junweil,alex}@cs.cmu.edu, {lujiang,lijiali}@google.com, liangliang.cao@gmail.com

Abstract

Recent insights on language and vision with neural net-

works have been successfully applied to simple single-

image visual question answering. However, to tackle real-

life question answering problems on multimedia collections

such as personal photos, we have to look at whole collec-

tions with sequences of photos or videos. When answering

questions from a large collection, a natural problem is to

identify snippets to support the answer. In this paper, we

describe a novel neural network called Focal Visual-Text

Attention network (FVTA) for collective reasoning in visual

question answering, where both visual and text sequence in-

formation such as images and text metadata are presented.

FVTA introduces an end-to-end approach that makes use of

a hierarchical process to dynamically determine what me-

dia and what time to focus on in the sequential data to an-

swer the question. FVTA can not only answer the questions

well but also provides the justiﬁcations which the system re-

sults are based upon to get the answers. FVTA achieves

state-of-the-art performance on the MemexQA dataset and

competitive results on the MovieQA dataset.

1. Introduction

Language and vision have emerged as a popular re-

search area in computer vision. Visual question answer-

ing (VQA) [2] is a successful direction utilizing both com-

puter vision and natural language processing techniques to

solve an interesting problem: given a pair of image and a

question (in natural language), the goal is to learn an in-

ference model that can the answer questions according to

cues discovered from the image. A variety of methods have

been proposed to address the challenges from different as-

pects [5, 27, 14, 6, 20, 3, 16, 13], with remarkable progress

on answering about a single image.

Extending from VQA on a single image, this paper con-

siders the following problem: Suppose a user’s photos and

videos are organized in a sequence ordered by their creation

time. Some photos or videos may be associated with meta

labels or annotations such as time, GPS, captions, com-

ments, and meaningful title. We are interested in training

Figure 1. Focal Visual-Text Attention (FVTA) Mechanism. Given

the visual-text sequences input and the question, our temporal

visual-text attention tensor captures the temporal constraint in the

question and emphasizes the most recent image with ”bar” scene

visible. Then FVTA selects the appropriate attention region (the

“date”) and ﬁnds the correct answer.

a model to answer questions about these images and texts,

e.g. “when was the last time I went to a bar?” or “what did

my son do after his 2017 Halloween dinner party?”

There are two challenges to solve the above problem.

First, the input is provided in an unstructured form. The

question is associated with multiple sequences, in the form

of videos or images. Such sequences are temporally or-

dered, and each sequence contains multiple time steps. At

each time there are visual data, text annotations and other

metadata. In this paper, we call the format visual-text se-

quence data. Note that not all the photos and videos are

annotated, which requires a robust method to leverage in-

consistently available multimodal data.

The second challenge requires interpretable justiﬁcations

in addition to direct answer based on sequence data. To help

users with a lot of photos and videos, a natural requirement

is to identify the supporting evidence for the answer. An

example question as shown in Fig. 1, is “when was the last

time I went to a bar?” From the users’ viewpoint, a good QA

system should not only give a deﬁnite answer (e.g., January

20, 2016), but also ground evidential images or text snippets

in the input sequence to justify the reasoning process. Given

arXiv:1806.01873v1 [cs.CV] 5 Jun 2018

imperfect VQA models, humans often want to verify the

answer. The inspection process may be trivial for a single

image but can take a signiﬁcant amount of time to examine

every image and the complete text words.

To address these two challenges, we propose a focal

visual-text attention (FVTA) model for sequential data

Our model is motivated by the reasoning process of humans.

In order to answer a question, a human would ﬁrst quickly

skim the input and then focus on a few, small temporal re-

gions in the visual-text sequences to derive an answer. In

fact, statistics suggest that, on average, humans only need

1.5 images to answer a question after the skimming [9]. In-

spired by this process, FVTA ﬁrst learns to localize relevant

information within a few, small, temporally consecutive re-

gions over the input sequences, and learns to infer an an-

swer based on the cross-modal statistics pooled from these

regions. FVTA proposes a novel kernel to compute the at-

tention tensor that jointly models the latent information in

three sources: 1) answer-signaling words in the question, 2)

temporal correlation within a sequence, and 3) cross-modal

interaction between the text and image. FVTA attention al-

lows for collective reasoning by the attention kernel learned

over a few, small, consecutive sub-sequences of text and

image. It can also produce a list of evidential images/texts

to justify the reasoning. As shown in Fig. 1, the highlighted

cubes are regions of high activations in the proposed FVTA.

To summarize, the contribution of this paper is threefold:

• We propose a novel attention kernel for VQA on

visual-text data. Experiments show that it outperforms

existing attention methods.

• The proposed attention tensor can be used to localize

evidential image and text snippets to explain the rea-

soning process. We quantitatively verify that the evi-

dence produced by our method are more correlated to

that of human annotators.

• Our method achieves the state-of-the-art results on two

VQA benchmarks.

2. Related Work

Visual Question Answering. Image-based visual ques-

tion answering has received a large amount of interest in the

computer vision community. A lot of efforts have been con-

ducted on single image QA datasets [2, 12, 31, 17, 26, 1],

where a common practice is to train a classiﬁer by combin-

ing both question feature and visual features. A recent di-

rection is on the question answering based on videos, which

is more relevant to this work. A number of research stud-

ies have been carried on MovieQA [22, 10, 15], with movie

clips, scripts, and descriptions. Because it is expensive to

Code and models are released at https://memexqa.cs.cmu.

edu/fvta.html

annotate the video-based QA datasets, some research stud-

ies generate QA datasets by harvesting online videos and

descriptions [30, 29], while a recent study [7] considers

question answering using animated GIFs. This work dif-

fers from the existing video-based QA in two aspects: (1)

video-based QA is to answer questions based on a single

video, while our work can handle general visual-text se-

quences, where one user may have more than one video or

albums of photos. (2) most existing video-based QA meth-

ods map one video sequence with text into a context feature

vector, while our work explores a more ﬁne-grained model

by modeling the correlation between query and sequence

data at every time step. To this end, we experiment on the

MemexQA dataset [9]. The sequential data in MemexQA

involves multiple modalities, including titles, timestamps,

GPS and visual content, render it an ideal test bed for QA

research over visual-text sequence data. Unlike the model

in [9], our method also uses the text embedding of the an-

swer choices as the input to answer a question.

Attention Mechanism. This work can be viewed as a

novel attention model for multiple variable-length sequen-

tial inputs, to take into account not only the visual-text in-

formation but also the temporal dependency. Our work ex-

tends the previous studies of using attention model for Im-

age QA [20, 4, 26, 13, 27, 16, 5, 3]. A key difference

between our method and classical attention model lies in

the fact we are modeling the correlation at every time step,

across multiple sequences. Existing attention mechanisms

for VQA mainly focus on attention within spatial regions of

an image [31] or within a single sequence [7], and hence,

may not fully exploit the multiple sequences and multiple

time steps nature. As Fig. 3 shows, our attention is applied

to a three-dimensional tensor, while the classic soft atten-

tion model is applied to a vector or matrix.

3. Approach

3.1. Problem Formulation

We start the discussion by formally deﬁning the prob-

lem. Let Q = q

, · · · , q

represent a question of M words

Q ∈ Z

, where each word is an integer index in the vocab-

ulary. Deﬁne a context visual-text sequence of T examples

X = x

, · · · , x

, where for each example, x

img

represents

an image. x

txt

is its corresponding text sentence, where its

i-th word is indexed by x

txt

. Following [2, 31], the answer

to a question is an integer y ∈ [1, L] over the answer vocab-

ulary of size L. Given a collection of n questions and their

context sequences, we are interested in learning a model

maximizing the following likelihood:

argmax

i=1

log P (y

, X

; Θ) (1)

where Θ represents the model parameters. Given the visual-

text sequence input X

img

, X

txt

, we obtain a good joint

of 10

50墨值下载

自动驾驶

关注

评论