Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation

0.4 MB·

by  Nasrin Mostafazadeh

Image-Grounded Conversations:
Multimodal Context for Natural Question and Response Generation
NasrinMostafazadeh1*,ChrisBrockett2,BillDolan2,MichelGalley2,JianfengGao2,
GeorgiosP.Spithourakis3*,LucyVanderwende2
1UniversityofRochester, 2Microsoft,
3UniversityCollegeLondon
[email protected], [email protected]
7
1
0
2

n
a Abstract
J
User1:Mysonisaheadandsurprised!
8
2 The popularity of image sharing on social User2:Didheendupwinningtherace?
media reflects the important role visual con- User1:Yeshewon,hecan’tbelieveit!
] text plays in everyday conversation. In
L
this paper, we present a novel task, Image-
C
Grounded Conversations (IGC), in which
Figure 1: A naturally-occurring Image-Grounded
s. natural-sounding conversations are generated
Conversation.
c about shared photographic images. We in-
[
vestigate this task using training data derived

1 from image-grounded conversations on so-
v cial media and introduce a new dataset of the image or answering questions about their con-
1 crowd-sourced conversations for benchmark- tent (Visual Question Answering (VQA)). In VQA,
5 ing progress. Experiments using deep neural questionsareconstrainedtobeanswerablefromthe
2
8 network models trained on social media data image,i.e.,theymightbeaskedbysomeoneunable
show that the combination of visual and tex-
0 toseetheimage. Understandinganimage,however,
1. tual context can enhance the quality of gen- involvesmorethancaptioningwhatisexplicitlyvis-
erated conversational turns. In human evalu-
0 ible. Figure1illustratesaconversationbetweentwo
ation,agapbetweenhumanperformanceand
7
that of both neural and retrieval architectures usersonsocialmedia. Theconversationisgrounded
1
: suggeststhatIGCpresentsaninterestingchal- not only in visible objects (e.g., the boys, the bike)
v
lengeforvisionandlanguageresearch. butmoreimportantly,ineventsandactions(e.g.,the
i
X race, winning) implied by the image. To humans
r viewingtheimages,thesemaybethemostinterest-
a
1 Introduction ingandmeaningfulaspects. VisualQuestionGener-
ation (VQG) (Mostafazadeh et al., 2016a) attempts
Significant advances in image captioning (Chen et
to address the challenge of how to generate ques-
al., 2015; Fang et al., 2014; Donahue et al., 2014;
tionsthatinvolvesuchcommonsenseunderstanding
Chen et al., 2015) have enabled much interdisci-
ofimagecontent.
plinaryresearchinvisionandlanguage,fromvideo
We extend VQG by introducing multimodal
transcription (Rohrbach et al., 2012; Venugopalan
conversational context when formulating questions
et al., 2015), to answering questions about images
around images and training on naturally occurring
(Antoletal., 2015; MalinowskiandFritz, 2014), to
socialmediadata. Tothisend,weintroducethetask
storytelling around series of photos (Huang et al.,
of Image-Grounded Conversation (IGC), which re-
2016). Much of the focus has been on understand-
quiresasystemtogeneratequestionsandresponses
ingimagesintermsofeitherdescribing(captioning)
in a natural-sounding conversation around a given
*ThisworkwasconductedatMicrosoft. image. IGC thus falls on the continuum between
chit-chatmodelsandgoal-directedconversationde- performance is improved by treating questions as a
signedtoaccomplishatask. Visualgroundinginan seriesinadialogratherthanseparateQApairs. This
image constrains the topic while providing objects formofdialogisbestconsideredasimpleone-sided
or inferrable events of interest so that a system can QA exchange, in which only humans can ask ques-
proactively drive the conversation forward. In this tionsandthesystemcanonlyprovideanswers. (Ray
paper we focus principally on generating questions et al., 2016) refine VQA by modeling whether the
asconversationdrivers. image contains enough information to answer the
This work thus draws together two threads of in- question; they observe that a model that can com-
vestigation that have hitherto remained largely un- ment on the answerability of the question is prefer-
related: vision & language and data-driven con- abletoasystemthatalwaysanswers.
versation models. The contributions of this paper Mostafazadehetal. (2016a)introducethetaskof
are threefold: (1) We extend VQG by introducing visualquestiongeneration(VQG),inwhichthesys-
multimodal conversational context when formulat- temitselfoutputsquestionsabouttheimage. Ques-
ing questions around images. To support this, we tions are required to be ‘natural and engaging’, i.e.
introduce the task of Image-Grounded Conversa- apersonwouldfindtheminterestingtoanswer,and
tion (IGC) via a crowd-sourced dataset of 4,222 6- may not be answerable from the image alone. In
turnimage-groundedconversationsthatwillbepub- this work, we build on Mostafazadeh et al. (2016a)
licly released, and compare IGC with other vision by introducing multimodal context when formulat-
& language tasks by analyzing the characteristics ingquestionsandresponses.
of our IGC datasets and the effect of multimodal
2.2 Data-DrivenConversationalModeling
context (Section 4). (2) We investigate the appli-
cation of deep neural generation and retrieval ap- This work is also closely linked to research on
proachesforquestionandresponsegenerationtasks data-driven conversation modeling. Ritter et al.
(Section 5), using models trained on 250K 3-turn (2011) posed the response generation as a machine
image-groundedconversationsfoundonTwitterand translation task, learning conversations from paral-
evaluatedonourcrowdsourceddataset. (3)Ourex- lel message-response pairs found on social media.
perimentssuggestthatthecombinationofvisualand Theirworkhasbeensuccessfullyextendedwiththe
textual context improves the quality of generated use of deep neural models (Sordoni et al., 2015;
conversational turns and that visual context is more Shanget al., 2015; Serban etal., 2015; Vinyalsand
important than textual (Section 8). It is our hope Le, 2015; Li et al., 2016a; Li et al., 2016b). Sor-
thatthisworkwillfurnishusefulbaselinestoothers donietal. (2015)introduceacontext-sensitiveneu-
workingonmultimodalconversationgeneration. rallanguagemodelthatselectsthemostprobablere-
sponseconditionedontheconversationhistory(i.e.,
2 RelatedWork a text-only context). In this paper, we extend the
contextualapproachwiththeadditionofmultimodal
2.1 VisionandLanguage
features to build models that are capable of asking
When trained on large datasets, such as the COCO questionsontopicsofintereststoahuman,andthat
dataset (Lin et al., 2014), Visual features combined might allow a conversational agent to proactively
with language modeling have shown good perfor- drivetheconversationforward.
mancebothinimagecaptioning(Devlinetal.,2015;
3 Image-GroundedConversations
Fang et al., 2014; Donahue et al., 2014) and in Vi-
sualQuestionAnswering(VQA)(Antoletal.,2015)
Forpresentpurposes,wedefinethescopeofIGCas
and (Malinowski and Fritz, 2014). In VQA, ques-
thefollowingtwoconsecutiveconversationalsteps:
tionsareconstrainedtobeanswerablefromtheim-
Question Generation: Given a visual context I
age,i.e.,theymightbeaskedbyapersonwhocannot •
and a textual context T (e.g., the first statement in
see the image. Das et al. (2016) extend the VQA
Figure 1), generate a coherent, natural question Q
scenario by collecting questions from people who
about the image as the second utterance in the con-
are shown only an automatically generated caption,
versation. AsseeninFigure1,thequestionmaynot
not the image itself, and demonstrate that system
bedirectlyanswerablefromtheimage.
ResponseGeneration: GivenavisualcontextI,a
•
textualcontextT,andaquestionQ,generateacoher-
ent, natural, response R to the question as the third
utteranceinthe conversation. The responsemaybe
an answer, as expected in the VQA or Visual Di-
alog tasks, or it may be a comment, deflection, or
other kind of response. The present work does not Figure2: ComparisonofV&Ldatasets.
attempt time to generate responses from generated
questions,ataskthatweleavetofuturework.
4 DataCollection
4.1 IGC
Twitter
Previousworkinneuralconversationmodeling(Rit-
teretal.,2010;Sordonietal.,2015)hassuccessfully
usedTwitterasthesourceofmillionsofnaturalcon-
versations. Inrecentyears,uploadingaphotoalong
with an accompanying tweet has become increas-
Figure 3: Distribution of the number of tokens
ingly popular: multimedia tweets have risen 15%
acrossdatasets.
per year (as of June 2015, 28% total), with 42%
of retweets containing non-verbal context (Morris photographgraphicsispotentiallyamajorsourceof
et al., 2016). For training data, we sampled 250K noiseinextractingfeaturesforneuralgeneration.
quadruplesof visualcontext,textualcontext,ques- We use the IGC dataset as primary training
{ Twitter
tion, response tweet threads from a larger dataset data. Forvalidationandtestsetsduringmodelbuild-
}
of 1.4 million, extracted from the Twitter Firehose ing, we held out a set of Twitter conversations, in
over a 3-year period beginning in May 2015 and whichimagesandconversationshadbeenvettedby
filtered to select just those conversations in which crowdworkerstobecontentfulandfreeofthekinds
the initial turn was associated with an image and ofimage-relatednoisenotedabove.
the second turn was a question. Regular expres-
sionswereusedtodetectquestions. Toimprovethe 4.2 IGCCrowd
likelihood that the authors are experienced Twitter To permit benchmarking of progress in the ICG
conversationalists, we further limited extraction to task,weconstructedtestandvalidationdatasetswith
those exchanges where users had actively engaged morecontrolledparametersonthebasisoftheVQG
in at least 30 conversational exchanges during a 3- dataset(Mostafazadehetal.,2016a). Wedesigneda
month period. Twitter data is notoriously noisy; crowdsourcing platform based on Turkserver (Mao
we performed simple normalizations, and filtered et al., 2012), which enables synchronous and real-
out tweets that contained mid-tweet hashtags, were time interactions between crowd workers on Ama-
longer than 80 characters1 and contained URLs not zon Mechanical Turk (Mturk). Multiple workers
linkingtotheimage. Table1presentsexamplecon- wait in a virtual lobby to be paired with another
versationsfromthisdataset. Althoughthefiltersre- workerwhowillbetheirconversationpartner. After
sult in significantly higher quality of the extracted beingpaired,oneoftheusersselectsanimagefrom
conversations, issues remain. A random sample of alargephotogallery,afterwhichthetwousersenter
tweets suggests that about 46% of the Twitter con- achatwindowinwhichtheyhaveashortconversa-
versationsisaffectedbypriorhistorybetweenusers, tionabouttheselectedimage.
makingresponsegenerationparticularlydifficult. In Images were sampled from the VQG dataset by
addition, the abundance of screen shots and non- querying a search engine using event-centric query
terms that aggregated ‘event’ and ‘process’ hy-
1Pilot studies showed that 80 character limit more effec-
ponyms in WordNet (Miller, 1995) and using fre-
tivelyretainsone-sentenceutterancesthataretothepoint.
Visual
Context
Textual Oh my gosh, i’m so buy- Ifoundacawaiibird. Stockingup!! OnlyreasonIcometocar-
Context ingthisshirt. nival.
Question Wheredidyouseethisfor Are you going to collect Ayee! what the prices OhmyGod. Howthehell
sale? somefeathers? lookinglike? doyoueveneatthat?
Response Midwestsports There are so many crows Only like 10-20% off..I They are the greatest
here I’d be surprised if I thinkI’mgonnawaitalit- things ever chan. I could
neverfoundone. tlelonger. eat5!
Table1: ExampleconversationsintheIGC dataset.
Twitter
Figure4: Distributionsofn-gramsequencesinquestionsinVQG,IGC ,andIGC .
Twitter Crowd
IGC (trainset) quent TimeBank events (Pustejovsky et al., 2003).
Twitter
#conversations=#images 250k Table 3 shows three full conversations found in the
total#utterances 750k IGC dataset. As the examples show, eventful
Crowd
IGC (valandtestsets,split:50%each)
Twitter imagesleadtoconversationswhicharesemantically
#conversations=#images 4653
total#utterances 13,959 very rich and would seem to require commonsense
IGC (valandtestsets,split:40%and60%) reasoning. Although the present work utilizes only
Crowd
#conversations=#images 4,222 threeconversationalturns,wecollecteduptosixut-
total#utterances 25,332
terancesperimageforuseinfuturework. Toenable
average#utterancesperconversation 4
multi-reference evaluation (Section 6), we crowd-
#allworkersparticipated 308
Max#conversationsbyoneworker 20 sourced four additional questions and responses for
Averageworktimeperworker(min) 9.5 the best IGC contexts and initial questions, as
Crowd
Medianworktimeperworker(min) 10.0
ranked by human annotators. The IGC dataset
Crowd
IGCCrowd−multiref (valandtestsets,split:40%and60%) willbepubliclyreleasedtotheresearchcommunity.
#additionalreferencesperquestion/response 5
total#multi-referenceutterances 42,220
4.3 DatasetCharacteristics
Table2: BasicDatasetStatistics. Table 2 summarizes basic dataset statistics. Fig-
ure2comparesIGCquestionswithVQGandVQA
questionsintermsofvocabularysize,percentageof
Visual
Context
Textual Thiswasn’tthewayIimaginedmy Icheckedouttheprotestyesterday. A terrible storm destroyed my
Context daystarting. house!
Question do you think this happened on the Do you think America can ever OHNO,whatareyougoingtodo?
highway? overcomeitsracialdivide?
Response Probably not, because I haven’t Icanonlyhopeso. IwillgolivewithmyDaduntilthe
driven anywhere except around insurancecompanysortsitout.
townrecently.
Turn4 Iwouldhavetohatetochangethe Wastheprotestpeaceful? it’s great that you can stay with
tireonthehighway. someone!
Turn5 Agreed, I should be grateful that I Yes,itwas,thankfully. Yeshewillbehappytohavemefor
noticeditbeforeIleftmyhome. awhile.
Turn6 Call AAA, they will change it for Ihopethevoicesofminoritieswill That’sgoodtohear.
you! be heard, and lead to changes in
policing.
VQG Whatcausedthattiretogoflat? Wherewastheprotest? What caused the building to fall
Question over?
Table3: ExamplefullconversationsinourIGC dataset. Forcomparison,wealsoincludeVQGques-
Crowd
tionsinwhichtheimageistheonlycontext.
abstract terms, and inter-annotation textual similar- annotation similarity. Figure 3 shows the distribu-
ity. The COCOimagecaptioning datasetis alsoin- tionofthenumberoftokenspersentence. Onaver-
cludedasapointofreference. TheIGC dataset age,theIGC datasethaslongersentences. Fig-
Twitter Twitter
has by far the largest vocabulary size, making it a ure 4 visualizes the n-gram distribution (with n=6)
morechallengingdatasetfortrainingpurposes. The of questions across datasets. IGC is the most
Twitter
IGC and IGC , in order, have the highest diverseset,withthelighter-coloredpartofthecircle
Crowd Twitter
ratioofabstracttoconcreteterms. Broadly,abstract indicating sequences with less than 0.1% represen-
terms refer to intangibles, such as concepts, quali- tationinthedataset.
ties, and feelings, whereas concrete terms refer to TheEffectivenessofMultimodalContext: The
things that can be experienced with the five senses. taskofIGCemphasizesmodelingofnotonlyvisual
It appears that conversational content may often in- butalsotextualcontext. Wepresentedhumanjudges
volveabstractconceptsthaneithercaptionsorques- witharandomsampleof600tripletsofimage, tex-
tionstargetingvisibleimagecontent. tual context, and question (I,T,Q) and asked them
It has been shown that humans achieve greater toratetheeffectivenessoftheimageandthetextual
consensus on what a natural question to ask given context, i.e., the degree to which the image or text
an image (the task of VQG) than on caption- isrequiredinorderforthesamplequestiontosound
ing or asking a visually verifiable question (VQA) natural. AsFigure5shows,overall,bothvisualand
(Mostafazadehetal.,2016b). Theright-mostplotin textualcontextsareindeedhighlyeffective,andun-
Figure2comparestheinter-annotationtextualsimi- derstandingbothwouldberequiredforthequestion
larity of our IGC questions using a smoothed thatwasasked. Wenotethatthecrowddatasetmore
Crowd
BLEU metric (Lin and Och, 2004). IGC often requires understanding of the textual context
Twitter
is excluded from this analysis as the data is not thantheTwittersetdoes.
multireference. Contextually grounded questions Frame Semantic Analysis: The grounded con-
of IGC are competitive with VQG in inter- versations with questions in our datasets are full of
Crowd
VisualContext Textual Con- Question
text
Look at all this Where is that
foodIordered! from?
FNFood Request-Entity Supplier
Table 4: FrameNet (FN) annotation of an example
triplet.
Figure 5: The effectiveness of textual and visual
contextforaskingquestions.
5.1 GenerationModels
Figure 6 overviews our three generation models.
stereotypical commonsense knowledge. To get a
The conversation shown is based on the first con-
bettersenseoftherichnessofourIGC dataset,
Crowd
versationinTable3.
we manually annotated a random sample of 330
(I,T,Q)tripletsintermsofMinsky’sFrames:2 We VisualContextSensitiveModel(V-Gen). Simi-
lartoRecurrentNeuralNetwork(RNN)modelsfor
annotated the FrameNet (Baker et al., 1998) frame
imagecaptioning(Devlinetal.,2015;Vinyalsetal.,
evoked by the image (I ), and then textual con-
FN
2015), (V-Gen) transforms the image feature vector
text (T ). Then, for the asked question, we anno-
FN
tated the frame slot3 (Q ) associated with a toa500-dimensionalvectorthatservesastheinitial
FN slot
− recurrentstatetoa500-dimensionalone-layerGated
contextframe(Q ). Theseannotationscanbeac-
FN
RecurrentUnit(GRU)whichisthedecodermodule.
cessed through https://goo.gl/MVyGzP. As
Theoutputsentenceisgeneratedonewordatatime
theexampleinTable4shows,theimageinisolation
until the <EOS> (end-of-sentence) token is gener-
oftendoesnotevokeanyuniquelycontentfulframe,
ated. Wesetthevocabularysizeto6000. Unknown
whereasthetextualcontextfrequentlydoes. Inonly
wordsaremapped toan<UNK>tokenduring train-
14% of cases does I =T , which further sup-
FN FN
ing,whichisnotallowedtobegeneratedatdecoding
ports the complementary effect of our multimodal
time.
contexts. Moreover, Q =I for 32% our anno-
FN FN
TextualContextSensitiveModel(T-Gen). This
tations,whereasQ =T for47%ofthetriplets,
FN FN
is a neural Machine Translation-like model that
againshowingtheeffectivenessoftextualcontextin
maps an input sequence to an output sequence
determiningthequestiontobeasked.
(Seq2Seq model (Cho et al., 2014; Sutskever et al.,
5 Models 2014)) using an encoder and a decoder RNN. The
decoder module is like the model described above,
We use the VGGNet architecture (Simonyan and
in this case the initial recurrent state being the 500-
Zisserman, 2014) for computing deep convolu-
dimensional encoding of the textual context. For
tional image features. We primarily use the 4096-
consistency, we use the same vocab size and num-
dimensionaloutputofthelastfullyconnectedlayer
beroflayersasinthe(V-Gen)model.
(fc7)astheinputtoallthemodelssensitivetovisual
Visual & Textual Context Sensitive Model
context.
(V&T-Gen). This model fully leverages both tex-
2Minskydefines‘frame’asfollows: “Whenoneencounters tualandvisualcontexts. Thevisionfeatureistrans-
anewsituation,oneselectsfrommemoryastructurecalleda formedtoa500-dimensionalvector,andthetextual
Frame” (Minsky, 1974). According to Minsky, a frame is a
context is likewise encoded into a 500-dimensional
commonsenseknowledgerepresentationdata-structureforrep-
vector. The textual feature vector can be obtained
resentingstereotypicalsituations,suchasaweddingceremony.
Minsky further connects frames to the nature of questions: usingeitherabag-of-words(V&T.BOW-Gen)repre-
“[AFrame] is a collection of questions to be asked about a sentation, or an RNN (V&T.RNN-Gen), as depicted
situation”. Thesequestionscanaskaboutthecause,intention,
in Figure 7. The textual feature vector is then con-
orside-effectsofapresentedsituation.
3For 17% of cases we could not find a corresponding catenated to the vision vector and fed into a fully
QFN slotinFrameNet. connected (FC) feed forward neural network. As
−
Figure6: QuestiongenerationusingtheVisualContextSensitiveModel(V-Gen),TextualContextSensitive
Model(T-Gen),andtheVisual&TextualContextSensitiveModel(V&T.BOW-Gen),respectively.
theN-bestlistusingthefollowingscorefunction:
log p(hC)+λidf(h,D)+µh +κV(h) (1)
| | |
wherep(hC)istheprobabilityofthegeneratedhy-
|
pothesis h given the context C. The function V
counts the number of verb POS in the hypothesis
and h denotesthenumberoftokensinthehypoth-
| |
esis. The function idf is the inverse document fre-
Figure 7: The visual & textual context sensitive
quency,simplycomputinghowcommonahypothe-
modelwithRNNencoding(V&T.RNN-Gen).
sisisacrossallthegeneratedN-bestlists. HereDis
a result, we obtain a single 500-dimensional vec- the set of all N-best lists and d is a specific N-best
D
tor encoding both visual and textual context, which list. We define idf(h,D) = log | | , where
d D:h d
then serves as the initial recurrent state of the de- we set N=10 to cut short each N|{-b∈est l∈ist}.|We op-
coderRNN. timizealltheparametersofthescoringfunctionto-
In order to generate the response (the third ut- wards maximizing the smoothed-BLEU score (Lin
terance in the conversation), we need to represent and Och, 2004) using the Pairwise Ranking Opti-
theconversationalturnsinthetextualcontextinput. mizationalgorithm(HopkinsandMay,2011).
There are various ways to represent conversational
history, including a bag of words model, or a con- 5.2 RetrievalModels
catenationofalltextualutterancesintoonesentence In addition to generation, we implemented two re-
(Sordoni et al., 2015). For response generation, we trieval models customized for the tasks of question
implement a more complex treatment in which ut- and response generation. Work in vision and lan-
terances are fed into an RNN one word at a time guagehasdemonstratedtheeffectivenessofretrieval
(Figure7)followingtheirtemporalorderinthecon- models, where one uses the annotation (e.g., cap-
versation. An <UTT> marker designates the end of tion) of a nearest neighbor in the training image set
oneutteranceandthebeginningofthenext. to annotate a given test image (Mostafazadeh et al.,
Decoding and Reranking. For all generation 2016a;Devlinetal.,2015;Hodoshetal.,2013;Or-
models, at decoding time we generate the N-best donezetal.,2011;Farhadietal.,2010).
lists using left-to-right beam search with beam-size Visual Context Sensitive Model (V-Ret). This
25. We set the maximum number of tokens to 13 modelonlyusesthegivenimageforretrieval. First,
forthegeneratedpartialhypotheses. Anypartialhy- we find a set of K nearest training images for the
pothesisthatreaches<EOS>tokenbecomesaviable given test image based on cosine similarity of the
fullhypothesisforreranking. Thefirstfewhypothe- fc7visionfeaturevectors. ThenweretrievethoseK
ses on top of the N-best lists generated by Seq2Seq annotationsasourpoolofK candidates. Finally,we
modelstendtobeverygeneric,4disregardingthein- compute the textual similarity among the questions
putcontext. Inordertoaddressthisissuewererank inthepoolaccordingtoaSmoothed-BLEU(Linand
Och, 2004) similarity score, then emit the sentence
4Anexamplegenericquestioniswhereisthis?andageneric
withthehighestsimilaritytotherestofthepool.
responseisIdon’tknow.
Visual
Context
on Textual The weather was amazing at this Igotinacarwrecktoday! Mycousinsatthefamilyreunion.
ati Context baseballgame.
er
n
e
G Gold Nice,whichteamwon? Didyougethurt? Whatisthenameofyourcousin
on Question intheblueshirt?
sti V&T-Ret Uatthegame?ordidsomeonetake Youdrivingthattoday? Uhadfun?
e
Qu thatpicforyou?
V-Gen Whereareyou? Who’sisthat? Who’sthatguy?
V&T-Gen Who’swinning? Whathappened? Where’smyinvite?
on Textual The weather was amazing at this I got in a car wreck today! Mycousinsatthefamilyreunion.
ati Context baseballgame.<UTT>Nice,which <UTT>Didyougethurt? <UTT>Whatisthenameofyour
ner teamwon? cousinintheblueshirt?
e
G Gold Myteamwonthisgame. Noitwasn’ttoobadofabangup. HisnameisEric.
se Response
n
o V&T-Ret 10formeand28formydad. Yes. lordscricketground.beautiful.
p
es V&T-Gen dingdingding! Nah,I’mathomenow. He’snotmine!
R
Table 5: Example baseline question and response generations on IGC test set. All the generation
Crowd
models usebeam searchwith reranking. Inthe textual context, <UTT> separates different utterances. The
generationsinboldareacceptableutterancesgiventheunderlyingcontext.
Visual & Textual Context Sensitive Model Althoughhumanevaluationistobepreferred,and
(V&T-Ret). Thismodelusesbothvisualandtextual currentlyessential,inopen-domaingenerationtasks
contextstoretrieveaquestionoraresponse. Alinear involving intrinsically diverse outputs, it is useful
combinationoffc7andword2vecfeaturevectorsis to have an automatic metric for day-to-day evalua-
utilizedforretrievingsimilartraininginstances. tion. For ease of replicability, we use the standard
MachineTranslationmetric,BLEU(Papinenietal.,
6 EvaluationSetup
2002), which captures n-gram overlap between hy-
pothesesandreferences. Resultsreportedbelowem-
We provide both human and automatic evaluations
ployBLEUwithequalweightsupto4-grams.
for our question and response generation tasks. We
crowdsourceourhumanevaluationonanAMT-like
7 ExperimentalResults
crowdsourcingsystem,askingsevencrowdworkers
toeachratethequalityofcandidatequestionsorre- We experiment with all the models presented in
sponses on a three-point Likert-like scale, ranging Section 5. For question generation, we use a vi-
from 1 to 3 (the highest). In order to ensure a cali- sual & textual sensitive model that uses bag-of-
bratedrating,weshowthehumanjudgesallsystem words(V&T.BOW-Gen)torepresentthetextualcon-
hypothesesforaparticulartestcaseatthesametime. text, which achieved better results. Earlier vi-
System outputs were randomly ordered to prevent sion & language work such as VQA (Antol et al.,
judgesfromguessingwhichsystemswerewhichon 2015) has shown that a bag-of-words baseline out-
thebasisofposition. Aftercollectingjudgments,we performsLSTM-basedmodelsforrepresentingtex-
averaged the scores throughout the test set for each tual input when visual features are available (Zhou
model. We discarded as spammers all annotators et al., 2015). In response generation, which needs
whose ratings varied from the mean by more than to account for textual input consisting of two turns,
2standarddeviations. we use the V&T.RNN-Gen model as the visual &
Human Generation(Greedy) Generation(Beam,best) Generation(Reranked,best) Retrieval
Gold Textual Visual V&T Textual Visual V&T VQG Textual Visual V&T Visual V&T
Twitter 2.26 1.39 2.13 1.57 1.06 2.35 1.90 1.49 1.03 1.72 1.71 1.71 1.68
Q.
Crowd 2.68 1.46 1.58 1.86 1.07 1.86 2.28 2.24 1.03 2.06 2.13 1.59 1.54
Twitter 2.44 1.26 – 1.60 1.13 – 1.68 – 1.05 – 1.60 – 1.59
R.
Crowd 2.75 1.24 – 1.40 1.12 – 1.49 – 1.04 – 1.44 – 1.48
Table 6: Human judgment results. The maximum score is 3. The Q. rows correspond to the question
generationtaskandtheR.rowscorrespondtoresponsegeneration. Permodel,thehumanscoreiscomputed
by averaging across multiple images. The boldfaced numbers show the highest score among the systems
andunderlinesignifiestheoverallhighestscore.
Generation Retrieval generation system scores higher than human on the
Textual Visual V&T VQG Visual V&T Twitter test set for question generation, but other-
Q. Twitter 0.94 2.06 2.13 0.37 0.61 0.75 wisethehumangoldreferencesinourCrowdsetare
Crowd 0.65 0.9 1.25 1.23 0.44 0.40
consistently favored throughout the table. We take
Crowdm 1.71 3.23 4.41 8.61 0.76 1.16
Twitter 0.35 – 0.44 – – 0.24 thisasevidencethatIGCCrowd providesarobustand
R.
Crowd 0.0 – 0.29 – – 0.15 challenging test set for benchmarking the progress
Crowdm 1.34 – 1.57 – – 0.66 onthetask.
BLEUscoresarelow, asischaracteristicforlan-
Table 7: Results of evaluating various models ac-
guage tasks with intrinsically diverse outputs (Li et
cording to automatic metric. Crowd refers to the
m al., 2016b; Li et al., 2016a). Automatic evaluation
multi-referencetestset.
in Table 7 provides confirmation that the IGC
Crowd
test set is more challenging to models that have
been trained on IGC . On BLEU, the multi-
textual-sensitivemodelintheR.rowsoftables6and Twitter
modalV&T modeloutperformsalltheothermodels
7. Since generating a response solely from the vi-
acrosstestsets,exceptforthemultireferencetestset
sual context is unlikely to be successful, we do not
(Crowd inTable7),inwhichtheVQGmodeldoes
usetheV-Genmodelinresponsegeneration. m
significantly better. We attribute this to differences
Table5presentsafewexamplegenerationsbyour
in the Twitter and our Crowd datasets, as discussed
bestperformingsystems. Tables6and7providethe
inSection4.
humanandautomaticevaluationresultsforallofour
Overall,inbothautomaticandhumanevaluation,
models. All models are trained on IGC (train-
Twitter
ourquestiongenerationmodelsaremoresuccessful
ingset),exceptforthemodellabeledVQG,whichis
than response generation. More sophisticated mod-
thesame(V-Gen)model,buttrainedon7,500ques-
els and larger training data sets may overcome this
tions from the VQG dataset (Mostafazadeh et al.,
disparityinthefuture.
2016b). All systems have been tuned/tested on the
correspondingIGCdataset. Asapointofreference,
8 Conclusions
weincludethegoldstandardhumanreferenceinthe
humanevaluations. We have introduced a new task of multimodal
Inhumanevaluation,themodelthatencodesboth image-groundedconversation,inwhich,whengiven
visual and textual context outperforms others, ex- an image and a natural language text, the system
ceptforthequestiongenerationonTwitterinwhich must generate meaningful conversation turns. To
the visual model wins. It appears the visual con- support this task, we are releasing to the research
text is more effective in Twitter dataset, as we have community a crowdsourced dataset of 4,222 high-
shown in Section 4. We note that human judges qualityconversationsabouteventfulimageswithup
preferred the top generation in the n-best list over to6turnseach,andmultiplereferences.
the reranked best, likely due to tradeoff between a Our experiments provide evidence that capturing
safeandgenericutteranceandariskierbutcontent- multimodal context improves the quality of genera-
ful one. As shown in Table 6, our best performing tion. Thegapbetweentheperformancesofourbest
models and humans opens opportunities further re- Hao Fang, Saurabh Gupta, Forrest N. Iandola, Ru-
searchinthecontinuumfromcasualconversationto pesh Srivastava, Li Deng, Piotr Dolla´r, Jianfeng
more task- and topic-oriented vision and language Gao, Xiaodong He, Margaret Mitchell, John C. Platt,
C. Lawrence Zitnick, and Geoffrey Zweig. 2014.
dialog. We expect also that addition of other kinds
From captions to visual concepts and back. CoRR,
of grounding may further improve performance of
abs/1411.4952.
systems.
AliFarhadi,MohsenHejrati,MohammadAminSadeghi,
Peter Young, Cyrus Rashtchian, Julia Hockenmaier,
andDavidForsyth. 2010. Everypicturetellsastory:
References
Generatingsentencesfromimages. InProceedingsof
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- the 11th European Conference on Computer Vision:
garetMitchell,DhruvBatra,C.LawrenceZitnick,and Part IV, ECCV’10, pages 15–29, Berlin, Heidelberg.
Devi Parikh. 2015. VQA: Visual question answer- Springer-Verlag.
ing. InInternationalConferenceonComputerVision Micah Hodosh, Peter Young, and Julia Hockenmaier.
(ICCV). 2013. Framing image description as a ranking task:
CollinF.Baker, CharlesJ.Fillmore, andJohnB.Lowe. Data,modelsandevaluationmetrics. J.Artif.Int.Res.,
1998. The berkeley framenet project. In Proceed- 47(1):853–899,May.
ingsofthe17thInternationalConferenceonCompu- MarkHopkinsandJonathanMay. 2011. Tuningasrank-
tational Linguistics - Volume 1, COLING ’98, pages ing. InProceedingsofthe2011ConferenceonEmpir-
86–90, Stroudsburg, PA, USA. Association for Com- ical Methods in Natural Language Processing, pages
putationalLinguistics. 1352–1362,Edinburgh,Scotland,UK.,July.Associa-
Jianfu Chen, Polina Kuznetsova, David Warren, and tionforComputationalLinguistics.
YejinChoi. 2015. De´ja` image-captions: Acorpusof Ting-Hao (Kenneth) Huang, Francis Ferraro, Nasrin
expressive descriptions in repetition. In Proceedings Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Ja-
ofthe2015ConferenceoftheNorthAmericanChapter cob Devlin, Ross Girshick, Xiaodong He, Pushmeet
oftheAssociationforComputationalLinguistics: Hu- Kohli,DhruvBatra,C.LawrenceZitnick,DeviParikh,
manLanguageTechnologies,pages504–514,Denver, Lucy Vanderwende, Michel Galley, and Margaret
Colorado, May–June. Association for Computational Mitchell. 2016. Visualstorytelling. InProceedingsof
Linguistics. the 2016 Conference of the North American Chapter
Kyunghyun Cho, Bart Van Merrie¨nboer, Caglar Gul- oftheAssociationforComputationalLinguistics: Hu-
cehre, Dzmitry Bahdanau, Fethi Bougares, Holger man Language Technologies, pages 1233–1239, San
Schwenk, and Yoshua Bengio. 2014. Learn- Diego, California, June. Association for Computa-
ing phrase representations using rnn encoder-decoder tionalLinguistics.
for statistical machine translation. arXiv preprint Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao,
arXiv:1406.1078. andBillDolan. 2016a. Adiversity-promotingobjec-
AbhishekDas,SatwikKottur,KhushiGupta,AviSingh, tivefunctionforneuralconversationmodels. InPro-
Deshraj Yadav, Jose´ M. F. Moura, Devi Parikh, ceedings of the 2016 Conference of the North Amer-
and Dhruv Batra. 2016. Visual dialog. CoRR, ican Chapter of the Association for Computational
abs/1611.08669. Linguistics: Human Language Technologies, pages
Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, 110–119,SanDiego,California,June.Associationfor
LiDeng,XiaodongHe,GeoffreyZweig,andMargaret ComputationalLinguistics.
Mitchell. 2015. Languagemodelsforimagecaption- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao,
ing: The quirks and what works. In Proceedings of andBillDolan. 2016b. Apersona-basedneuralcon-
the53rdAnnualMeetingoftheAssociationforCom- versationmodel. InProceedingsofthe2015Confer-
putational Linguistics and the 7th International Joint ence of the North American Chapter of the Associa-
ConferenceonNaturalLanguageProcessing(Volume tionforComputationalLinguistics:HumanLanguage
2:ShortPapers),pages100–105,Beijing,China,July. Technologies.AssociationforComputationalLinguis-
AssociationforComputationalLinguistics. tics.
Jeff Donahue, Lisa Anne Hendricks, Sergio Guadar- Chin-Yew Lin and Franz Josef Och. 2004. Automatic
rama, Marcus Rohrbach, Subhashini Venugopalan, evaluationofmachinetranslationqualityusinglongest
KateSaenko,andTrevorDarrell. 2014. Long-termre- common subsequence and skip-bigram statistics. In
current convolutional networks for visual recognition Proceedingsofthe42NdAnnualMeetingonAssocia-
anddescription. CoRR,abs/1411.4389. tionforComputationalLinguistics,ACL’04,Strouds-