Balancing Novelty and Salience: Adaptive Learning to Rank Entities for Timeline Summarization of High-impact Events

1.3 MB·

by  Tuan Tran

Balancing Novelty and Salience:
Adaptive Learning to Rank Entities for Timeline
Summarization of High-impact Events
Tuan Tran, Claudia Niederée, Nattiya Kanhabua, Ujwal Gadiraju, and Avishek Anand
L3SResearchCenter/LeibnizUniversitätHannover,Germany
{ttran, niederee, kanhabua, gadiraju, anand}@L3S.de
7 ABSTRACT Germanwings Germanwings Andreas Lubitz
01 Long-running,high-impacteventssuchastheBostonMarathon Fcrlaisghhedt i9n5to2 th5e Alp F“cdlriaegstahhi ltoe fd9 G o5Ff2 v45icU ti9m5s2 5in … th e “acLbonul-aabpcliytiklzosb itdso Ae xcln iobddnearfertiaaram tsee lyd…
2 bombing often develop through many stages and involve a Digne-les-Bain Joseph-König- Carsten Spohr
n lraizragteionnumofbearnoefveennttitibeysikneythseeirntuennfcoeldsienags.eTsismtoerlyindeisguemstmiona-, “ahacocrrcoriisbdslee 5n” ta scitree ss p..r”e, a“ids G“accofltyanesfrmi sr1me6nse sadct sautondiu cehenmaltlvsee d daite JdK G Lscstrpuuaefnstehhncaeehndd sa …atbh oCa. utgE tci Ovo…e -ps. ilao t
a butdoesnotdistinguishbetweenwhatauserremembersand Germanwings Francois Hollande Joseph-König-
J several Germanwings President FH: “a tragedy Gymnasium
what she might want to re-check. In this work, we present flight cancelled after on our soil” ….. report A moment of silence
crew refused to fly from President Francois is held Thursday at
4 anovelapproachfortimelinesummarizationofhigh-impact Hollande conflicts with…. JKG
1 events, which uses entities instead of sentences for summa- 24 March 25 March 26 March

rizingtheeventateachindividualpointintime. Suchentity Figure 1: Entity Timeline for the 2015 Germanwings
]
R summariescanserveasboth(1)importantmemorycuesin Plane Crash Event
aretrospectiveeventconsiderationand(2)pointersforper-
I
. sonalizedeventexploration. Inordertoautomaticallycreate
s
such summaries, it is crucial to identify the“right”entities we study event timeline summarization and present a novel
c
[ for inclusion. We propose to learn a ranking function for method which shows key entities at different time points of
entities, with a dynamically adapted trade-off between the an event thus capturing this dynamic event unfolding. In

1 in-document salience of entities and the informativeness of contrast to other work in event summarization [34, 23], our
v entities across documents, i.e., the level of new information entity timelines use entities instead of sentences as main
7 associated with an entity for a time point under considera- units of summarization as depicted in the case of the 2015
4 tion. Furthermore, for capturing collective attention for an Germanwings plan crash (Figure 1). Such summaries can
9
entityweuseaninnovativesoftlabelingapproachbasedon be easily digested and used both as starting points for per-
3
Wikipedia. Our experiments on a real large news datasets sonalized exploration of event details, and for retrospective
0
confirm the effectiveness of the proposed methods. revisiting. The latter can be triggered by a new similar
.
1 event, or by a new twist in the story. For example, the tes-
CategoriesandSubjectDescriptorsH.3.3[InformationStor-
0 timonial of the captain in the Costa Concordia trial in late
7 ageandRetrieval]: InformationSearchandRetrieval 2014 triggered a revisiting of the disaster in 2012.
1 General TermsAlgorithms,Experimentation Fromacognitiveperspective,foreventrevisiting,werather
: KeywordsEntityRetrieval,TimelineSummarization,Wikipedia, create“memorycues”tosupportrememberingtheunfolded
v LearningtoRank,News,TemporalRanking
i events than summaries for rehearsing the details. In fact,
X memorycuescanberegardedas“acircumstanceorpieceof
r 1. INTRODUCTION informationwhichaidsthememoryinretrievingdetailsnot
a High-impact, real-world events often unfold in an unex- re-called spontaneously” (Oxford online dictionary, 2015).
pectedway,dynamicallyinvolvingavarietyofentities. This In this sense, our work is related to the idea of designing
isespeciallytrueforlong-runningeventswhenfullinforma- orcreatingmemorycuesforreal-liferemembering[29]. En-
tionabouttheeventanditsdevelopmentbecomesavailable tities such as persons and locations have been identified as
only in the course of days after the happening, as in the veryeffectiveexternalmemorycues[2]. Inaddition,theim-
caseofarecentGermanwingsairplanecrash. Inthispaper, portance of entities in event summarization has also been
shown in recent work [24, 25].
For creating an entity timeline,the entities to be used in
Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalor
thesummaryhavetobechosencarefully. Theyshould1)be
classroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributed
characteristicforarespectivetimepointoftheevent,2)not
forprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcita-
tiononthefirstpage. Copyrightsforcomponentsofthisworkownedbyothersthan be repetitive (if nothing new happened with respect to the
ACMmustbehonored.Abstractingwithcreditispermitted.Tocopyotherwise,orre- entities),3)beassociatedtorelevanteventinformation,and
publish,topostonserversortoredistributetolists,requirespriorspecificpermission 4) be interesting to the reader. For this purpose, we pro-
and/[email protected]. pose an approach to entity summarization, which dynami-
CIKM’15,October19-23,2015,Melbourne,VIC,Australia.
cally combines entity salience with the informativeness of
Copyright2015ACM978-1-4503-3794-6/15/10...$15.00
entities at a considered point in time. Entity salience, on
http://dx.doi.org/10.1145/2806416.2806486.
the one hand, considers the property of being in the focus help improving the performance of document retrieval [25].
of attention in a document has been studied in previous Thereisabodyofworkinidentifyingsaliententitiesingen-
work [5, 12, 10]. In [5], Boguraev and Kennedy use salient eralWeb[12]andinnewsdomain[10],buttheexistingwork
text phrases for the creation of so-called capsule overviews, does not take into account the time dimension. Our work
whereas recently methods for the identification of salient targets a different question, using entities as a pivot, and it
entities, e.g., in Web pages [12] and news articles [10], have considerstheglobalcontext(timelineoftheevent)asanew
been developed. Informativeness, on the other hand, as- dimension of entity salience determination.
sessesthelevelofnewinformationassociatedwithanentity Learning to Rank Multiple Criteria. In our work, we
in a text and can be computationally measured using fea- apply a learning to rank (L2R) technique to generate time-
turesderivedfromstatisticalandlinguisticinformation[33]. line summaries in a similar way as done in [28, 23]. In con-
In more detail, we aim at optimizing a trade-off between trasttothepreviouswork,weproposeanoveljointlearning
thein-documentsalienceofentitiesandtheinformativeness method, which optimizes different aspects of an event. The
of entities across documents describing the unfolding of an model is able to distinguish the aspects semantically. We
event. Our contributions are: also incorporate the time effect (i.e., decay) into the joint
model,makingourlearningtoranktime-sensitiveandsuit-
• Wearethefirsttoproposeanentitytimelinefornovel
able for timeline summarization.
user experience in event exploration and digestion.
• Weproposeanovelentityrankingmodelthatdynam-
3. OVERVIEW
ically learns a trade-off between entity salience and
informativeness.
3.1 Preliminaries
• For training our ranking model, we propose a method
Long-running Event. Following [23], we define a long-
for gathering soft labels (or ground truths) by mining
runningeventas“anewsworthyhappeningintheworldthat
collective attention in Wikipedia.
is significant enough to be reported on over multiple days”.
Wevalidateourcontributionswithexperimentsonalarge Each event is represented by a short textual description or
real-world news dataset covering various types of events. asetofkeywordsq,wherewewilluseq todenotetheevent
The rest of the chapter is organized as follows. Section 2 intherestofthispaper. Forexample,thebombingincident
discusses the related work. Sections 3,4 details our pro- during Boston Marathon in April 2013 can be described by
posedmethod. Section5discussesfeaturesusedinlearning theterms“bostonmarathonbombing”. Weassumethatthe
framework. Section 6 presents the experiments and results. relevant time frame is split into a sequence of consecutive
Finally, section 7 concludes our paper. non-overlapping, equal-sized time intervals T ={t1,...,tn}
in a chronological order, in our case individual days. Fur-
2. RELATEDWORK thermore, for a given event q, there is a set of timestamped
documents D (each with a publication date) reporting on
q
News Summarization. The task of news summarization
theevent. Wedefineareportingtimeline T ={t ,...,t }
has been already studied in various contexts, which range q k1 kj
asanorderedlistof(notnecessarilyconsecutive)thosetime
from focusing on multi-document summarization [5, 11] to
periods t in T, which contain the publication date of at
generatingatimelinesummaryforaspecificnewsstory[34, ki
leastdocumentinD . Finally,wedenotethesetofalldoc-
q
23, 28, 35]. News stories can be complex, having a non-
uments about q published within a time period t as D .
linear structure and associated to multiple aspects. Shahaf ki q,i
etal.[27]proposeanovelmethodforsummarizingcomplex Entity. We are interested in named entities mentioned in
storiesusingmetromapsthatexplicitlycapturetherelations documents, namely, persons, organizations, locations. An
amongdifferentaspectsanddisplayatemporaldevelopment entityecanbeidentifiedbyacanonicalname,andcanhave
of the stories. Instead of using documents or sentences as a multiple terms or phrases associated with e, called labels,
information units, we provide a set of entities, as memory whichrefertotheentity. Wecallanappearanceofanentity
cues,forsupportingeventexplorationanddigestionateach label in a document, a mention of e, denoted by m. We
individualpointintime. Tothebestofourknowledge,none defineE asthesetofallentitiesmentionedinD . Further-
q q
of the previous work provides a trade-off solution that bal- more, we define the text snippet surrounding m (e.g., sen-
ances between content-based and collective attention-based tence or phrases) as the mention context, denoted c(m).
approaches,insupportingtheentity-centricsummarization.
Entity Ranking and Entity Salience. In recent years, Entity Salience and Informativeness. Similarlyto[10],
entityretrievalandrankinghavegainedinterestfromtheIR we define the entity salience as the quality of“being in the
community moving beyond traditional document retrieval focus of attention” in the corresponding documents. An-
to more specific information about entities in response to a other relevant aspect considered for selecting entities to be
query, e.g., [24, 25]. Although a lot of interesting work on includedinaneventtimelineis informativeness [12],which
entity ranking has already been proposed, most of previous imposes that selected entities in an evolving event should
works have focused on static collections, thus ignoring the also deliver novel information when compared to the past
temporaldynamicsof queriesanddocuments. Therelevant information. For example, although the airline ”German-
workclosesttooursinthisrespectisbyDemartinietal.[8], wings”staysrelevantformanyarticlesreportingontheplane
where the task tackled is to identify the entities that best crash,itwillonlybeconsideredasinformative,ifnewinfor-
describe the documents for a given query. The entities are mation about the airline becomes available.
identifiedbyanalyzingthetop-kretrieveddocumentsatthe
time when the query was issued as well as relevant docu- Problem. Given a long-running event q, a time interval t
i
mentsinthepast. InIR,saliententitiesinnewsdocuments in its reporting timeline T , and the set of entities E , we
q q
aimtoidentifythetop-k salientandinformativeentitiesfor Event-­‐relevant

  SoE
 Labeling

supporting the exploration and digestion of q at t . documents
  using
 Wikipedia

i
e

3.2 ApproachOverview Informa)ve
  Co-­‐training
 Phas
einn-tWditoeicetusa,mcwkehlneitcthehniitssitpayrimosbaelldeiemnactbeoyaplnteidamrtnihziienngignaftohfruemntacrttaiidovene-nofeoffsrsbroeaftnwekneineting- tt-­‐
 1q

 ….d
 oc
  reCfoeNCnrEeotRne-­‐
 xcte
 s
 En))es
 features
  LeJaorinnitn
 g
  Salient
  admapoo$dfv
 eael

 Training
 ase

ties across documents: Extrac)on
  Model

  (q,t)
  Ph
Adapta)on
  n

t+1
  o
where y(q) is theyvt(eqc)t=orfo(fEr,aωnsk,iωnig),sfco∈reFs for entities in(E1) ….
 Dq
  Eexnt$rtayc
 $&o
 cno
 ntext
  features
  ERnasnemkinbgle

  rq,t(e)
  foR$ram
 enenkl$iinntgey

 Applica$
t q
attimeintervalt,Eisamatrixcomposedfromfeaturevec-
Figure2: OverviewoftheEntity-centricSummarization
torsofentitiesinE extractedfromtheirmentioncontexts.
q Framework
ω ,ω are the unknown parameter vectors for ranking enti-
s i
ties based on salience and informativeness, respectively.
In our work, a ranking function is based on a learning-
4. METHODOLOGY
to-rank technique [19]. A general approach for learning-to-
rank is to optimize a defined loss function L given manual
annotationorjudgmentsy(q) ofentitiesforasetoftraining 4.1 EntityExtraction
j
events Q within the time intervals Tq: First,wediscusshowwecanextractentities,thekeyunits
fˆ=argmin(cid:88) (cid:88) L(f(E(q),ω ,ω),y(q)) (2) inourframework. Foreachdocumentintheset,,weemploy
j s i j a named entity recognizer to identify mentions of entities
f∈F
q∈Qtj∈Tq in three categories (persons, locations and organizations).
Two major challenges must be taken into account when For each mention, we include the containing sentences as
learningarankingfunctiondefinedinEquation2. First,we thecontext. Mentionsthatdonotcontainanyalphabetical
needareliablesourceforbuildingjudgments(groundtruths) characters or only stop words are removed.
forannotatingentitiesbyconsideringtheirsaliencewithre- Weadditionallyuseintra-andcross-documentco-reference
spect to a given event. In addition, the judgments must be to track mentions pertaining to the same entity. First, an
dynamically adapted to the evolving of entities along the intra-document co-reference system is employed to identify
unfolding event, i.e., bearing of novel information. Second, all co-reference chains for entity mentions within a docu-
themodelsofourtwoaspectsω ,ω mustbeunifiedtopro- ment. We include each reference in the chain together with
s i
duce a joint learned function for ranking entities. In the its sentence to the set of mention contexts of the entity.
subsequent sections, we will explain our proposed method Second, to identify mentions that potentially refer to the
for these challenges in more detail. samereal-worldentityacrossdocuments,weadaptthestate-
of-the-art cross-document co-reference method proposed by
3.3 Framework
Leeetal.[22]. Thismethodfirstclustersdocumentsbasedon
Figure 2 gives an overview of our entity ranking frame- their content using an Expectation Maximization (EM) al-
workcoveringboththetrainingandtheapplication/testing gorithm,theniterativelymergesreferences(verbalandnom-
phase. inal) to entities and events in each cluster.
Given one event q, its reporting timeline, and the set of Speeding up co-reference resolution: To speed up
documentsD (inpractice,D canbegivenapriori,orcan the computation, we do not use EM clustering as [22], but
q q
beretrievedusingdifferentretrievalmodels)weidentifythe employ a set of heuristics, which have proven to be effec-
entity set E using our entity extraction, which consists of tive in practice. First, we only consider cross-document co-
q
named entity recognition, co-reference and context extrac- referencesfromdocumentsofthesameday. Second,instead
tion (Section 4.1). of clustering an entire document set, we use mentions with
When the event is used for training (training phase), we their contextual sentences (kept in the order of their ap-
link a subset of E to Wikipedia concepts, which comprises pearanceintheoriginaldocuments)as“pseudo-documents”
q
the popular and emerging entities of the event. To facil- for the clustering. Third, we assume that mentions to the
itate the learning process, these entities are softly labeled same entity have similar labels. Hence, we represent en-
using view statistics from Wikipedia (Section 4.2), serving titymentionlabelsasvectorsusingtwo-grams(forinstance,
as training instances. Although we use popular entities for “Obama”becomes“ob”,“ba”,“am”,“ma”) and apply LSH
training,wedesignthefeaturessuchthatitcanbegeneral- clustering [15] to group similar mentions. LSH has been
ized to arbitrary entities, independent from Wikipedia. proven to perform well in entity disambiguation tasks [17],
The next component in our framework is the adaptive anditismuchfasterthanthestandardEM-basedclustering.
learningthatjointlylearnsthesalienceandinformativeness Finally, for each entity mention, we merge its contextual
models,takingintoaccountthediversenatureofeventsand sentences with those of all other references of the same co-
their evolution. (Section 4.4). referencechaintoobtaintheevent-levelcontextforanentity
Intheapplicationphase,entityandfeatureextractionare , which will be used as inputs for constructing the entity
appliedthesameasintrainingphase. First,theinputevent features. Wenotethatwhileweareawareofothermethods
and time interval is examined against the joint models to to increase the quality of entity extraction by linking them
return the adaptive scores (details in Section 4.5). Then, to a knowledge base such as YAGO or DBpedia, we choose
entitiesareputintoanensembleranking,usingtheadapted not to limit our entities to such knowledge bases, so as to
models, to produce the final ranks for the summary. beabletoidentifyandrankmoreentitiesinthe“longtail”.
4.2 MiningWikipediaforSoftLabeling and time t , we rank an entity e higher than others if: (1)
i
Inthefollowing,weexplainhowthetrainingdataisgen- e is more relevant to the central parts of documents in Dq,i
erated. Specifically, given an event q, an interval t and an whereitappears(salience);and(2)thecontextofeismore
entity e, we aim to automatically generate the score y(q) diverse to other contexts (of e or other entities) at Dq,i−1.
j Moreover,thesetwocriteriashouldbeunifiedinanadaptive
suchthaty(q)(e)>y(q)(e(cid:48))iftheentityeismoreprominent
j j waythatdependsonthequery,thatis,eventandtime. For
thantheentitye(cid:48) withrespecttotheeventqattimet. This example, users interested in a festival might wish to know
score can be used as a soft labelling to learn the ranking more about salient entities of the event, while those that
functionsmentionedinEquation2. Theuseofsoftlabeling follow a breaking story prefer entities with more fresh in-
forentities’saliencehasalreadybeenproposedin[12],where formation. Even for one event, the importance of salience
userclickbehaviourinquerylogsisusedasanindicatorfor and informativeness might vary over time. For instance,
entity salience scores. Dunnietz et al. [10] proposed treat- informativeness is more important at the beginning when
ingentitiesinnewsheadlinesassalient,andpropagatethose theeventisupdatedfrequently. Basedonthisintuition,we
saliencescorestootherentitiesviathePageRankalgorithm. propose the following ranking function:
Thelimitationofthesemeasuresisthattheyrestricttheas-
sessment of salience to the scope of individual documents, y(q) =S(q,t)ωTE + γ(t)I(q,t)ωTE (4)
t s s i i
and do not consider the temporal dimension. In contrast,
whereE isthe|E |×M matrixrepresentingtheM dimen-
oursoftlabelsareevolving,i.e.,anentitycanhavedifferent s q
sional feature vectors of entities used to learn the salience
labels for one event at different time intervals.
score (M is the number of salience features), and E is the
Thesoftlabelingisbasedontheassumptionthatforglob- i
|E|×N isthematrixofN dimensionalinformativenessfea-
allytrendingnewsevent,prominenceofrelatedentitiescan
ture vectors (N is the number of informativeness features).
be observed by the collective attention paid to resources
S(q,t) and I(q,t) represent the scores of salience and infor-
representing the entities. For instance, during the Boston
mativeness tendency for an event q at t. Here we introduce
marathonbombing,WikipediapagesaboutthebomberTsar-
another factor, γ(t), which is the decay function of time t,
naev were created and viewed 15,000 times after one day,
controlling how much the informativeness should have im-
indicatingtheirstrongsaliencedrivenbytheevent. Forsoft
pactontheoverallranking. Therationaleofγ isthatwhen
labeling,weexploitthepageviewstatisticofWikipediaar-
thedistancebetweentwotimeintervalst andt 1 islong,
ticles,whichreflectstheinterestofusersinanentity: Most i i−1
informativeness has less impact on the overall ranking. For
obviously, Wikipedia articles are viewed for currently pop-
example, if there are only reports about the news after one
ular entities indicating entity salience. However, taking the
year (anniversary of a past event), the changes of entities
encyclopediccharacterofWikipediaintoaccount,Wikipedia
in that long time period should not contribute much to the
articles are also viewed in expectation of new information
informativeness criterion.
about an entity indicating its (expected) informativeness,
especially in the context of an ongoing event. Actually,
4.4 Multi-criteriaLearningModel
Wikipedia has gained attention in recent years as a source
of temporal event information [32]. Event-triggered bursts We now discuss how to learn the above ranking function
in page views, as they are for example used in [7] for event using Equation 2. A straightforward way is to learn the
detection, are thus a good proxy for the interestingness of two models ωs and ωi separately, and assign values to S,I
anevent-relatedentityat aconsideredpointintime, which in a pre-defined manner, then aggregate into Equation 4.
isinfluencedbothbythesalienceandtheinformativenessof However, this is notdesirable, since itrequires tobuild two
the event. sets of training data for salience and informativeness at the
Therefore, we propose a new metric called View Outlier sametime,whichisexpensive. Secondly,previousworkhas
Ratio orVORtoapproximatethesoftlabelsforacombined pointed out that a“hard”classification of query based on
measure of entity salience and informativeness as follows. intentitselfisadifficultproblem,andcanharmtheranking
For each entity e and for a given time interval t , we first performance [14]. In this work, we exploit the divide and
i
constructthetimeseriesofviewcountfromthecorrespond- conquer (DAC) learning framework in [3] as follows. We
ing Wikipedia page of e in the window of size w: define E∗ as the |E|×(M +N) matrix of (M +N) dimen-
sional extension vectors from the corresponding vectors of
Te =[ve,i−w,ve,i−w+1,...ve,i,ve,i+1,...vi+w] E andE matrices. Similarly,wedefineω∗ asthe(M+N)
s i s
where each ve,j is the view count of the Wikipedia page of extension vectors of zero vector 0, and the vectors ωs, and
e at tj. From Te we calculate the median me,i and define ωi∗ as the (N +M) extension vector of ωi and 0. With this
VOR as follows. transformation, Equation 4 can be written as:
Definition 1. TheViewOutlierRatioistheratioofdif- y(q)=S(q,t)g(E∗,ω∗) + γ(t)I(q,t)g(E∗,ω∗) (5)
t s i
ference between the entity view and the median:
|v −m | where g(E∗,ω) = ωTE∗ is a linear function. Incorporating
vor(ei)= max(em,ie,i,me,miin) (3) Equations 2 and 5, we can co-learn the models ωs∗,ωi∗ (and
thus ω ,ω ) simultaneously, using any loss functions. For
s s
where m is a minimum threshold to regularize entities
min instance,ifweusehingelossasin[3],wecanthenadaptthe
with too low view activity.
RankSVM [19], an algorithm that seeks to learn the linear
4.3 UnifiedRankingFunction function g in (5) by minimizing the number of misordered
We now turn our attention to defining the ranking func- 1notethatnewseventsarenotalwaysreportedonconsecu-
tion in Equation 1. The intuition is that for each event q tive time intervals
document pairs. For completeness, we describe here the impact onto the informativeness of entities: When the time
traditional objective function of RankSVM: lag between two intervals is high, the difference in contexts
min 1(cid:107)ω(cid:107)2+c (cid:88) ξ , s.t. of entities between them is less likely to correlate with the
ω,ξq,t,a,b 2 q,t,a,b q,t,a,b (6) informativeness quality of entities.
ωTE∗(q)≥ωTE∗(q)+1−ξ ∀E∗(q)(cid:31)E∗(q),ξ ≥0
t,a t,b q,t,a,b t,a t,b q,t,a,b
5. ENTITYRANKINGFEATURES
where E∗(t,qa) (cid:31) E∗(t,qb) implies that entity a is ranked higher We now discuss the salience and informativeness features
than entity b for the event q at time t, ξq,t,a,b denotes slack for entity ranking. Ranking features are extracted from
variables,andcsetsthetrade-offbetweenthetrainingerror eventdocumentswheretheentityappearsasfollows. These
andmodelcomplexity. Ifwechangethelinearfunctiong to features, called individual features, are extracted two differ-
f,wecanadapt(2)into(6)toobtainthefollowingobjective ent levels. First, at context level, features are extracted
function: independentlyfromeachmentionanditscontexts. Features
min (cid:107)ωs(cid:107)2+(cid:107)ωi(cid:107)2 +c (cid:88) ξ s.t. of this level include mention word offset, context length, or
ω,ξq,t,a,b 2 q,t,a,b q,t,a,b importance scores of the context within the document us-
S(q,t)g(E∗(t,qa),ωs∗)+γ(t)I(q,t)g(E∗(t,qa),ωi∗)≥ (7) itnugressu).mmSeacroinzadt,ioant alalgboerlitlhemvesl,(SfeuamtuBraessicaroereSxutmraFcotecdusfrfoema-
S(q,t)g(E∗(t,qb),ωs∗)+γ(t)I(q,t)g(E∗(t,qb),ωi∗)+1−ξq,t,a,b, all mentions, for instance aggregated term (document) fre-
∀E∗(q)(cid:31)E∗(q),ξ ≥0 quencies of mentions.
t,a t,b q,t,a,b Based on the individual features, the entity features are
constructed as follows. For each entity and feature dimen-
4.5 Event-basedModelsAdaptation sion,wehavethelistoffeaturevalues(z1,z2,...,zn),where
z is the individual feature of label or mention categories.
TheadaptivescoresS(q,t),I(q,t)andthedecayfunction i
For label level, we simply take the average of z ’s over all
γ(t) is critical to adapting the salience and informativeness i
entity labels. For mention level, each z is weighted by the
models. A na¨ıve supervised approach to pre-define the cat- i
the confidence score of the document containing the corre-
egories for event (S,I) is impractical and detrimental to
sponding mention and context. Such confidence score can
ranking performance if the training data is biased. Instead,
be calculated by several ways, for instance by a reference
previous work on query-dependent ranking [14, 3] often ex-
model (e.g., BM25) when retrieving the document, or by
ploit the “locality property”of query spaces, i.e., features
calculating the authority score of the document in the col-
of queries of the same category are more similar than those
lection (e.g., using PageRank algorithm). For all features,
of different categories. Bian et al.[3] constructed query fea-
we apply quantile normalization, such that all individual
turesusingtop-retrieveddocuments,andclusteredthemvia
features(andthusentityaggregatedfeatures)arescaledbe-
a mixture model. However, the feature setting is the same
tween[0,1]. Belowwedescribethemostimportantfeatures.
for all clusters, making it hard to infer the semantics of the
query categories. 5.1 SalienceFeatures
In this work, we inherit and adjust the approach in [3]
Context importance features. One important evidence
as follows. For each event q and time t, we obtain all en-
ofentitysalienceiscontextoftheentitymentions. Itiswell-
tities appearing in D to build the“pseudo-feedback”for
q,t
knownthattextatthebeginningofdocumentcontainsmore
the query (q,t). We then build the query features from the
salientinformation[12,10]. Besidesposition,thecontentof
pseudo-feedback as follows. From each matrix E ,E, we
s i
sentences, per se or in relations with other sentences, also
take the mean and variance of the feature values of all en-
indicates the salience of entities. We apply SumBasic [26]
tities in the pseudo feedback. As a result, each pair (q,t)
and LexRank [11] summarization algorithms to obtain the
is mapped into two feature vectors (with 2M and 2N di-
scores of contexts (features Sum-B and PR, respectively).
mensions)correspondingtothesalienceandinformativeness
spaces. In each space, we use Gaussian Mixture model to
Human-perceived salience features. Entity salience
calculate the centroid of the training queries, and use the
can be assessed by reader’s intentions or interests [12], and
distance of the query feature vector to the centroid as its
recentstudysuggeststhatuserinterestinentitiescanbeat-
corresponding adaptive score:
(cid:107)xq,t−xC(cid:107)2 tracted via serendipity in texts [6]. We follow this direction
C(q,t)=1− C (8) andapplyfeaturespresentedin[6],namelysetimentality,at-
maxq(cid:48)∈Q,t(cid:48)∈Tq(cid:48)(cid:107)xqC(cid:48),t(cid:48)−xC(cid:107)2 titudes,andreadabilityscoresofeachmentioncontext. For
readability,wealsoincludetwootherstandardmetrics,Gun-
where C ∈ {I,S} indicates the event categories, xq,t is the
C ning Fog index [16] and Fleisch-Kincaid index [20].
query feature in the feature space of C, and xC is the cen-
troidoffeaturevectorsintrainingsetQinthecorresponding Query-related features. Another class of salience fea-
space. The scores are scaled between 0 and 1. turesinvolvesonesthataredependentontheeventqueries.
Following[31],weusetheoverlapbetweencontextsandthe
Decayed Informativeness. The decay function γ(t ) ad-
i event query at the unigram, bigram levels, and at bigram
justs the contribution of informativeness into the adaptive
where tokens are tolerable to have 4 words in between (Bi4
model and is defined by:
feature). Wealsocomputethequery-focusedSumFocus[30]
γ(ti)=αλ|ti−µti−1| (9) scoreofcontextsasanotherfeature. Itisworthnotingthat
for all of these features, the event queries used are the ones
where λ,α are parameters (0 < α < 1,λ > 0), and µ is that have been extended using the query expansion (see
the interval unit distance. Equation 9 represents the time Event Document and Timelines, Section 6.1).
Salience features Informativeness features
Feature(s) Description Feature(s) Description
Tf/Df(M) Term/Doc. frequencyofmentioninDq,i PTf/Pdf(M) Term/Doc. frequencyofmentioninDq,i−1
WO/SO(C) Word/sent. offsetofmentionincontext CoEntE(M) Numberofco-occurringentitiesinDq,i−1
SentLen(C) Contextlengthwith/withoutstopwords CTI(C) CTIscoreofmentiongivenitscontextinDq,i
Sent-5/-10(C) Context length with/without stopwords > andDq,i−1 [33]
5/10?
1-/2-/3-Sent(C) Iscontextamong1/3/5firstsentences? TDiv(C,M) TopicdiversityofcontextinDq,i,Dq,i−1
TITLE(M) Isthementionintitlesofanyd∈Dq,i ? CosSim(C) CosinesimilarityofcontextinDq,i,Dq,i−1
CoEntM(M) No. ofentitiesinsamecontextasmention Distributional similarity of context in
DisSim(C)
Sum-B/-F(C) SumBasic/SumFocusscoreofcontext Dq,i,Dq,i−1
Uni/Bi/Bi4(C) Uni-/Bi-/skip-Bigramoverlapwithquery EntDif(M) EntitydifferenceofcontextinDq,i,Dq,i−1
Att/Sent(C) Attitude/Sentimentalityscoreofcontext TITLEP(M) Isthementionintitlesofanyd∈Dq,i−1 ?
Read-1/-2/-3(C) Fleisch/Fog/Kincaidreadabilityscores NewsFracP(M) Frac. ofnews-specifictermsinDq,i−1
PR(C) Sentencecentralityscore(PageRank)
POS(C) POS-tagofmentionincontext
NewsFrac(M) Fractionofnews-specifictermsco-occurring
withmention
Table 1: Selected Informativeness and Salience Features for Entity Ranking at Label (M) and Context (C) Level
5.2 InformativenessFeatures tion features over D for event q at time t (Table 1).
q,i−1 i
For label-level topic diversity, we use the same metric, but
Informativenessforwordshasbeenstudiedextensivelyin
usingconcatenatedcontextsinsteadofindividualones. Ad-
linguistics community. We adapt the state-of-the-art mea-
ditionally, we also calculate the entity difference between
sure[33],whichincorporatesthecontextofmentions. Given two context sets of the same entity in D and D :
q,i−1 q,i
a mention m of entity e, the context-aware informativeness
score of m for event q at ti is defined by: EntDif(e)=(cid:107)CoEnti(e)∩CoEnti−1(e)(cid:107)
 1 Uq,i−1=∅ whereCoEnti(e)issetofco-occurringentitiesofeinDq,i.
inf(m,i)= (cid:80)c(cid:48)∈Uq,i−1κ|(Ucq(cid:48),,ic−(m1|))s(dc(cid:48),q,i−1) Uq,i−1(cid:54)=∅ 6. EXPERIMENTSANDEVALUATIONS
where Uq,i−1 is the set of mention contexts of e in Dq,i−1, 6.1 Setup
dc(cid:48) is the document consisting the context c(cid:48), κ(c(cid:48),c(m)) is Datasets. Forourexperiments,weworkwithareal-world,
thesentencedis-similarity,ands(dc(cid:48),q,i−1)istheretrieval large-scale news dataset. Specifically, we use KBA 2014
score of dc(cid:48). We normalize the metric to [0,1], and set it Filtered Stream Corpus (SC14) dataset, which is used for
to 1, when the entity first appears in D . To measure
κ(c(cid:48),c(m)), we can employ different stratqe,giies, leading to TREC 2014 Temporal Summarization track2. We extract
news and mainstream articles from the dataset, consisting
different features:
of7,592,062documents. Thedatasetcovers15long-running
CTI.Themoststraightforwardstrategyistoemployalexi- news events from December 2012 to April 2013. All events
conforthesentencesimilarity. WeusetheNESimmethod[9] are high-impact, discussed largely in news media, and have
for this strategy. their own Wikipedia pages. Each event has a predefined
time span (ranging from 4 to 18 days), and is represented
TopicDiversity. Besideslexicon-based,wealsocalculatethe byonetextualphrasethatisusedastheinitialeventquery.
informativeness on a higher level by representing contexts Basedontheeventtype(availableinthedataset),wegroup
bylatenttopics,usinglatentDirichletAllocationmodel[4]. theeventsinto4categories: Accident,riotandprotest,nat-
Topic diversity of a context c w.r.t other context c(cid:48) of the ural disaster, crime (shooting, bombing).
same entity on previous time interval is defined as
Event Documents. To construct the event document set
(cid:118)
(cid:117) τ for the study, we firstly group the documents into individ-
κ(c(cid:48),c(m))=(cid:117)(cid:116)(cid:88)(p(ψk|c(m))−p(ψk|c(cid:48)))2 ual days by their publication timestamps, and index docu-
k=1 mentsforeachday. Intotal, thisresultsin126differentin-
dices. Foreachindex,weremoveboilerplatetextsfromdoc-
where τ is the number of topics and ψ is the topic index.
k umentusingBoilerpipe[21],skipstopwords,andlemmatize
Distributional Similarity. Another strategy uses Kullback- the terms. Then we use the pre-defined textual phrases of
Leibler divergence for the dissimilarity: the events, issue it as a query to the corresponding indices
DisSim(c(cid:48),c(m))=−KL(θc(m),θc(cid:48))= (10) (eiancdhictehseotfodpa1y0s wdoitchuimnetnhtes euvseinngt pBeMri2o5d)w, eriegthrtieinvgingmofrdoeml.
−(cid:88)wi P(wi|θc(m))logPP(w(wi|iθ|cθ(c(cid:48)m)))) (11) Wsioenim[1p8r],ovaendthaedrdestuoltps3u0sinexgpKanudllebdactke-rLmeisbtleorcqounesrtyruecxtptahne-
event query q used for query-related features computation.
where θd and θc are the language models for contexts c(cid:48),c. Timelines. To build the reporting timeline Tq, for each
We use Dirichlet smoothing method for scarce words. event,wemanuallygothroughallthedaysoftheeventpe-
Label Level. Besides context level, we also calculate a 2http://s3.amazonaws.com/aws-publicdatasets/trec/
number of features at label level, by aggregating the men- kba/index.html
riod, check the content of the top-retrieved document, and by incrementally selecting sentences, maximizing the gain
remove the day from the timeline if this top-ranked docu- and coverage with respect to summaries on previous days.
ments is not about the event. In total, we have 153 pairs Since we are not interested in adaptively determining the
(event,day) for all event reporting timelines. cutoffvaluesforthesummary,weimplementonlythelearning-
Entities. We use Stanford parser to recognize named en- to-rank method reported in [23] to score the sentences. We
tities from the boilerplated content of the documents, and adapt IUS into entity summarization by extracting named
match them with the entities detected by BBN’s Serif tool entities from each sentence. Then, the ranking score of the
(providedinSC14corpus)toreducenoise. Forthematching entity is calculated as the average of scores of all of its sen-
entities, we use the in-document co-reference chains, which tences across all documents.
are available in SC14, and apply the cross-document coref- In addition, we evaluate three other variants of our ap-
erence (Section 4.1) to group mentions to entities. We use proach. The first two variants involve only salience and in-
the sentences as the mention contexts. In total, we detect formativenessfeaturesforlearning. WedenotetheseasSAL
72,267 named entities from the corpus, with an average of and INF. The third variant linearly combines all salience
5.04 contexts per each. and informativeness features, denoted as No-Adapt. All are
Training Data. From 153 (event,day) pairs, we randomly trained and predicted using the traditional RankSVM.
choose 4 events belonging to 4 different categories men- EvaluationMetrics. Weconsiderthetraditionalinforma-
tioned above as a training data, resulting in 39 pairs. To tion retrieval performance metrics: precisions, NDCG and
build training entities (i.e. to identify subset of entities to MAP. Besides, we also aim to evaluate the performance of
Wikipedia concepts, see Section 4.1), we apply two named ranking in timeline summarization context, where effective
entitydisambiguationsoftwares,WikipediaMinerandTagme. systemsdonotjustintroducerelevant,butalsonovelandin-
These are the supervised machine learning tools to identify terestingresultscomparedtothepast. Thiswasinspiredby
named entities from natural language texts and link them recentworkinuserexperienceofentityretrieval[13,6]. One
to Wikipedia. We train the models of both the tools from popular metric widely adopted in existing work to measure
a Wikipedia dump downloaded in 2014 July, so as to cover such user-perceived quality is the serendipity, which mea-
allpossibleentitiesintheSC14corpus. Weonlyuseentities sures degree to which results are“not highly relevant but
co-detectedbyboththetools,resultingin402distinctenti- interesting”to user taste [1]. Traditional serendipity metric
ties and 665 training tuples (entity,event,day). We use the wasproposedtocontrasttheretrievalresultswithsomeob-
Wikipediapageviewdataset,whichispubliclyavailable,to viousbaseline[6]. Inourcase,weproposetouseserendipity
build the soft labels for these entities. to measure the informativeness and salience by contrasting
Parameter Settings. We modify RankSVM for our joint the results of one day to previous day of the same event:
learning tasks. Features are normalized using the Stan-
(cid:80)
dard scaling. We tune parameters via grid search with 5- SRDP = e∈UNEXP rel(e) (12)
fold cross validation and set trade-off parameter c = 20. |UNEXP|
WikipediaMiner and Tagme tools are used with default pa-
where UNEXP is the set of entities not appearing on the
rametersettings. Forthedecayparameters,giventherather
previous day, and rel is the human relevance judgment of
smalltimerangeofeventsinourdataset,weempiricallyset
the entity. The relevance part ensures the salience of the
λ=2,α=0.5,µ=1day,andleavemoreadvancetuningfor
entity,whiletheUNEXP partensurestheinformativeness
future work. For soft labeling, we set the window size w to
of the entity over the event reporting timeline.
10 days, which is the average length of reporting timelines.
The threshold m is tuned as followed. For each training AssessmentSetup.
min
pair, a human expert knowing the 4 training events well is We exclude the 39 training pairs from the overall 153
presentedwiththeentities,theirmentioncontextsandcon- (event,day) pairs to obtain 114 pairs for testing. For each
tentofthecorrespondingdocument. Theexpertisaskedto of these pairs, we pooled the top-10 entities returned by all
put the labels on the entity from“falsely detected”, “non- methods. Intotal,thisresultsin3,336tuples(entity,event,
salient entity”,“salient but not informative”to“salient and day) to be assessed. To accommodate the assessment, we
informative”. Based on this judgement, we compute VOR contextualize the tuples as follows. For each tuple, we ex-
scores with mmin from 1 to 100, and optimize the rank of tractonesentencecontainingtheentityfromthedocument
entities based on VOR scores using NDCG metric. We find with the highest retrieval score (BM25), using the event
that mmin =12 yields the best performance. as the query and on the index corresponding to the day.
If there are several sentences, we extract the longest one.
6.2 Evaluation Next, we describe our two assessments setup, expert-based
and crowdsource-based.
Baselines. We compare our approach with the following
Expert Assessment. To evaluate the quality of the sys-
competitive baselines.
tems, we employ an expert-based evaluation as follows. 5
TAER:Dermatinietal. [8]proposedalearningframework
volunteers who are IT experts and work on temporal and
to retrieve the most salient entities from the news, taking
event analysis were asked to assess on one or several events
into consideration information from documents previously
of their interest. For each event, the assessors were encour-
published. This approach can be considered as “salience-
agedtocheckthecorrespondingWikipediapagebeforehand
pro”, since the entity salience is measured within a docu-
to gain sufficient knowledge. Then, for each tuple, we add
ment, although it implicitly complements the informative-
one more contextualizing sentence, extracted from the pre-
ness via history documents. We train the model on the
viousdateoftheevent. Ifthereisnosuchsentence,a“NIL”
same annotated data provided by the author.
stringwillbepresented. Weaskedtheassessorstocheckthe
IUS[23]: This work represents the“informativeness-pro”
tuple and the two sentences, and optionally, to use search
approach,itattemptstobuildupdatesummariesforevents
enginestolookformoreeventinformationonthequestioned the isolated features can only be achieved by fusing the
date. Then, the assessors were asked to assess the impor- two features in a more sophisticated way: No-Adapt does
tance of the entity with respect to the event and date, in not perform better than the maximum of SAL and INF
four following scales. 1: Entity is obviously not relevant to (MAX(S,I)),itevenperformsworseinundersomemetrics
the event; 2: Entity is relevant to the event, but it has no such as P@10. In contrast, AdaptER clearly outperforms
new information compared to the previous day; 3: Entity the maximum of SAL and INF as well as its non-adaptive
is relevant to the event and linked to new information, but version for most metrics. For instance, we achieve 16% im-
it does not play a salient role in the sentence; 4: Entity is provementofMAPscoresascomparedwiththeMAX(S,I).
relevanttotheevent,hasnewinformation,andissalientin Besidesprecision,wealsoconsiderserendipity(SRDP)as
the presented sentence. The inter-assessor agreement score a complementary measure in our experiments, as discussed
for this task is κ=0.4 under the Cohen’s Kappa score. above. Thismetricmeasureshowlikelytheapproachbrings
Crowdsourced Assessment. In addition, we also set up unseenandinterestingresultstotheuser. UnderSRDP,our
a larger-scale assessment based on crowdsourcing. We use approachoutperformssignificantlyboththebaselineandthe
Crowdflower.com platform to deploy the evaluation tasks. maximum of SAL and INF. We achieve 14% improvement
Each tuple presented to workers consists of date, short de- of serendipity at top-1 entities, and 29% at top-3 entities.
scription of the event, entity, and the sentence. Instead of Thus,ourtop-retrievedentitiesdonotonlycoverrelevance,
direct asking for salience and informativeness of entities to but are also more interesting, often unseen on the previous
event and date we decide for a simpler task: Asking the day (contributing to more informative results).
workerstoassessentitiesintwosteps: (1)Assessingwhether The lower part of the Table 2 shows the same results for
the sentence is obviously relevant to the event (workers as- the crowdsourced assessment. The same trends of perfor-
sess on a 3-point Likert scale,from“1-Not Relavant”to“3- mance can be observed, where our approach outperforms
Obviously Relevant”); and (2) assessing whether the entity in all the metrics. In comparison to the expert evaluation,
isimportantinthesentence(byvirtueofbeingasubjector theresultsareoveralllower. Apossiblereasonforthisisthe
object of the sentence, binary feedback). complexityofthecrowdsourcingtask,whichrequiresknowl-
Tasksaredeliveredsuchthattuplesofthesame(event,day) edgeabouttheconsideredeventinordertogivehighquality
pair go into one Crowdflower job, thus the worker has a feedback(seeExpertAssessmentsetup,Section6.1). Never-
chance to gain knowledge about event on the day and re- theless,theadaptivemodelisstillabletoachievesignificant
spond faster and more reliably. We pay USD 0.03 for each gain, especially under the serendipity measurement.
tuple. To maintain the quality, we follow state-of-the-art
guidelinesandrecommendations,andreceive5independent P@10 MAP
responses for eachtuple. We create agoldstandard for 311
No-Adapt Soft-labeled 0.32 0.225
tuples,anddiscardresponsesfromworkerswhofailtomain- Supervised 0.346 0.247
tain an agreement of above 70% against the gold standard.
AdaptER Soft-labeled 0.368 0.264
Intotal,wereceived20760responses,8940fromwhichwere Supervised 0.357 0.259
qualified. The inter-worker agreement was 98.67% under
Pairwise Percent Agreement, with average variance of 42%,
Table3: Performanceofsystemswith(-out)softlabeling
indicating a reasonably good quality given the fairly high
complexity of the task.
Next,weevaluatetheeffectivenessofsoftlabelingincov-
ering salience and informativeness. For this purpose, we
6.3 ResultsandDiscussion
manuallyannotateentitiesobtainedfromthereportingtime-
The upper part of Table 2 summarizes the main results line of 4 training events, with respect to the salience and
of our experiments from the expert evaluation. The results informativeness (see Parameter settings, Section 6.1). We
showtheperformanceofthetwobaselines(TAERandIUS) then re-train both non-adaptive and adaptive models using
andoftheconsiderationofSalienceandInformativenessfea- this annotated data (supervised approach). Table 3 shows
tures in isolation with respect to precision. In general, all theprecisionandMAPscoresofthesupervisedapproachin
performances are low, indicating the relatively high com- comparison to our soft labelling-based approach. The simi-
plexityofthisnewtask. Inaddition,ascanbeseenfromthis larity in performance between them, regardless of the mod-
part of the table, even the approach relying on our salience els they have been used for, confirms that our soft labelling
featuresorinformativenessfeaturesinisolationalreadyout- properly captures both salience and informativeness.
performsthetwobaselines. Thisisduetothefactthatour
approach does not consider documents in isolation as the FeatureAnalysis. Analysingtheinfluenceofdifferentfea-
baselines do. Rather, we take a more comprehensive view ture groups (see Section 5) can give insights into what fac-
considering event level instead of document level features torscontributetotheentityrankingperformance. Tostudy
via feature aggregation. In more details, the first baseline the feature impacts, we do an ablation study and remove
(TAER)employsaquiterestrictedfeaturesforentityrank- incrementally a group of features, and re-evaluate the per-
ing (e.g. document frequency), and thus fails to identify formance using the expert assessment. Since reducing the
important entities event-wise. featuredimensionsdirectlyaffectsthequeryfeaturesandits
Furthermore,theresultsalsoshowtheperformanceofthe adaptive scores, to ensure the fair comparison, we perform
non-adaptive combination of salience and informativeness the study for the non-adaptive setting. Table 4 shows the
(No-Adapt) as well as our approach (AdaptER), which uses MAP scores of ablated models. The symbol (cid:72) indicates a
an adaptive combination of informativeness and saliency. significant decrease with respect to No-Adapt model (with
It becomes clear that an improvement by combining the fullfeatures),andthusimpliesthehighinfluenceofthecor-
salience and the informativeness features over the use of responding feature group. From the table, we can see that
Method P@1 P@3 P@10 MAP SRDP@1 SRDP@3 SRDP@10
Ranking performance from expert assessment
TAER 0.436 0.315 0.182 0.109 0.315 0.210 0.121
IUS 0.395 0.325 0.236 0.141 0.335 0.217 0.176
SAL 0.493(cid:78) 0.423 0.338(cid:78) 0.217(cid:78) 0.421 0.320 0.240(cid:78)
INF 0.480(cid:78) 0.436 0.354(cid:78) 0.227(cid:78) 0.441(cid:78) 0.340 0.256(cid:78)
MAX(S,I) 0.493 0.436 0.354 0.227 0.441 0.340 0.256
No-Adapt 0.503 0.461 0.320 0.225 0.396 0.338 0.215
AdaptER 0.546 0.485 0.368 0.264 0.507(cid:78) 0.440(cid:78) 0.275
Ranking performance from crowdsourced assessment
TAER 0.229 0.183 0.106 0.066 0.201 0.146 0.079
IUS 0.258 0.202 0.154 0.092 0.197 0.165 0.119
SAL 0.320 0.279 0.207 0.139 0.279 0.218 0.154
INF 0.313 0.283 0.214 0.146 0.306 0.229(cid:78) 0.160
MAX(S,I) 0.320 0.283 0.214 0.146 0.306 0.229 0.160
No-Adapt 0.271 0.252 0.181 0.123 0.236 0.208 0.144
AdaptER 0.388(cid:78) 0.340 0.237 0.178 0.361 0.361(cid:78) 0.315(cid:78)
Table 2: Entity-ranking performance using different assessment settings. In each setting, significance is
tested against line 1, TAER (within the first group), and line 5, MAX(S,I) (within the second group).
Symbol (cid:78) indicates cases with confirmed significant increase
the most influential feature groups include context impor- thermore,weintroduceanadaptivelearningtorankframe-
tancefeature(saliencefeatures)andinformativenessfeature workthatjointlylearnsthesalienceandinformativenessfea-
group of context level. tures in a unified manner. To scale the learning, we exploit
Wikipedia page views as an implicit signal of user interest
Ablated feature set MAP towards entities related to high-impact events. Our exper-
iments have shown that the introduced methods consider-
Context importance 0.086(cid:72)
ably improve the entity selection performance, using both
Human-perceived 0.097
small-scale expert-based and large-scale crowdsourced as-
Informativeness, context level 0.06(cid:72)
sessments. The evaluation also confirms that integrating
Informativeness, label level 0.120
salience and informativeness can significantly improve the
user experience of finding surprisingly interesting results.
Table 4: MAP of No-Adapt when feature groups are
As the problem discussed in the paper is new, there are
ablated (full-feature MAP score: 0.225)
several promising directions to explore for future work. We
aim to investigate further the impact of adaptation in dif-
Anecdotic Example. In table 5, we show one example of
ferent types of events, using larger and more diverse sets
top-selected entities for the event“Boston marathon bomb-
of events. Furthermore, we plan to study more advanced
ing 2013”. Additionally, we show some selected sentences
ways to mine Wikipedia temporal information as signals of
coveringtheentities,toenabletheunderstandingoftheen-
collective attention towards public events. We are planning
tities’ roles within the event on the presented days. As can
to use this for further improving our VOR measure for the
beseen,thetimelinecorrespondingtoTAERapproach(up-
soft labeling approach. Other direction includes investigat-
per part) gives more salience credits to entities frequently
ing deeper models to improve the performances of current
mentionedthroughoutthenews(suchasBostonmarathon),
entitytimelinesummarizationsystems,whicharequitelow.
keeping them in high ranks throughout the timeline. The
approachisnotresponsivetolesssalientbutinterestingen- Acknowledgements. This work was partially funded by
tities (such as Pop Francis, a rather unrelated entity to the the European Commission for the FP7 project ForgetIT
event, but get involved via his condolecence and activities (under grant No. 600826) and the ERC Advanced Grant
to victims of the bombing). On the other hand, using an ALEXANDRIA (under grant No. 339233).
adaptiverankingwithinformativenessincorporated,there-
sulting entities are not just more diverse (including related 8. REFERENCES
eventssuchasMarathonBruins),butalsoexposemorenew [1] P.Andr´e,J.Teevan,andS.T.Dumais.Fromx-raystosilly
and emerging information. puttyviauranus: serendipityanditsroleinwebsearch.In
CHI,2009.
[2] D.Berntsen.Involuntary autobiographical memories: An
7. CONCLUSIONS introduction to the unbidden past.CambridgeUniversity
Press,2009.
In this paper, we have presented a novel approach for
[3] J.Bian,X.Li,F.Li,Z.Zheng,andH.Zha.Ranking
timeline summarization of high-impact news events, using
specializationforwebsearch: adivide-and-conquer
entities as the main unit of summary. We propose to dy- approachbyusingtopicalranksvm.InWWW,2010.
namicallyadaptbetweenentitysalienceandinformativeness [4] D.M.Blei,A.Y.Ng,andM.I.Jordan.Latentdirichlet
for improving the user experience of the summary. Fur- allocation.J. Mach. Learn. Res.,2003.
April15 April16 April17
Boston Marathon Boston Boston Marathon
Mass General Hospital Boston Marathon Boston
Boston.com Vatican Boston University
- Two bombs exploded near the Deeply grieved by news of the loss -FBIconfirmedthatpressurecook-
finish of the Boston Marathon on of life and grave injuries caused ersmayhavebeenusedasexplosive
Monday, killing two people, injur- by the act of violence perpetrated devicesattheBostonMarathon.
ing22others lasteveninginBoston,HisHoliness - The third victim was identified
- At least four people are in the Pope Francis wishes me to assure Wednesday as Boston University
emergency room at Mass General youofhissympathy... graduatestudentLingziLu.
Hospital
Boston Marathon Pope Francis FBI
Marathon Bruins Vatican Boston University
New York City Boston Marathon Lingzi Lu
- Two bombs exploded near the The Vatican sent a telegram to -FBIconfirmedthatpressurecook-
finish of the Boston Marathon on Boston Cardinal on Tuesday, in ersmayhavebeenusedasexplosive
Monday, killing two people, injur- whichPopeFrancisexpressessym- devicesattheBostonMarathon.
ing22others pathy for the victims of the - The third victim was identified
- The NHL postponed the Boston marathonbombings... Wednesday as Boston University
Bruins’ Monday hockey game due graduatestudentLingziLu.
tothebombing
Table 5: Examples of top-3 entities on Boston Marathon Bombing 2013 using TAER (top)
and AdaptER(bottom) for April 15, 16 and 17
[5] B.BoguraevandC.Kennedy.Salience-basedcontent [21] C.Kohlschu¨tter,P.Fankhauser,andW.Nejdl.Boilerplate
characterisationoftextdocuments.ACL,1997. detectionusingshallowtextfeatures.InWSDM.ACM,
[6] I.Bordino,Y.Mejova,andM.Lalmas.Penguinsin 2010.
sweaters,orserendipitousentitysearchonuser-generated [22] H.Lee,M.Recasens,A.Chang,M.Surdeanu,and
content.InCIKM,2013. D.Jurafsky.Jointentityandeventcoreferenceresolution
[7] M.CiglanandK.Nørv˚ag.Wikipop: Personalizedevent acrossdocuments.InEMNLP,2012.
detectionsystembasedonwikipediapageviewstatistics. [23] R.McCreadie,C.Macdonald,andI.Ounis.Incremental
InCIKM,2010. updatesummarization: Adaptivesentenceselectionbased
[8] G.Demartini,M.M.S.Missen,R.Blanco,and onprevalenceandnovelty.InCIKM,2014.
H.Zaragoza.Taer: time-awareentityretrieval-exploiting [24] X.Meng,F.Wei,X.Liu,M.Zhou,S.Li,andH.Wang.
thepasttofindrelevantentitiesinnewsarticles.InCIKM, Entity-centrictopic-orientedopinionsummarizationin
2010. twitter.InKDD,2012.
[9] Q.Do,D.Roth,M.Sammons,Y.Tu,andV.Vydiswaran. [25] Y.Moshfeghi,M.Matthews,R.Blanco,andJ.M.Jose.
Robust,light-weightapproachestocomputelexical Influenceoftimelineandnamed-entitycomponentsonuser
similarity.Computer Science Research and Technical engagement.InECIR.2013.
Reports, University of Illinois,2009. [26] A.NenkovaandR.Passonneau.Evaluatingcontent
[10] J.DunietzandD.Gillick.Anewentitysaliencetaskwith selectioninsummarization: Thepyramidmethod.In
millionsoftrainingexamples.EACL,2014. NAACL-HLT,2004.
[11] G.ErkanandD.R.Radev.Lexrank: Graph-basedlexical [27] D.Shahaf,C.Guestrin,andE.Horvitz.Trainsofthought:
centralityassalienceintextsummarization.J.Artif.Intell. Generatinginformationmaps.InWWW,2012.
Res.(JAIR),22(1):457–479,2004. [28] G.B.Tran,T.Tran,N.-K.Tran,M.Alrifai,and
[12] M.Gamon,T.Yano,X.Song,J.Apacible,andP.Pantel. N.Kanhabua.Leveragelearningtorankinanoptimization
Identifyingsaliententitiesinwebpages.InCIKM,2013. frameworkfortimelinesummarization.InTAIA Workshop
[13] M.Ge,C.Delgado-Battenfeld,andD.Jannach.Beyond at SIGIR,2013.
accuracy: evaluatingrecommendersystemsbycoverageand [29] E.vandenHovenandB.Egge.Thecueiskey-designfor
serendipity.InRecSys,2010. real-liferemembering.Zeitschrift fu¨r Psychologie.,
[14] X.Geng,T.-Y.Liu,T.Qin,A.Arnold,H.Li,andH.-Y. 222(2):110–117,2014.
Shum.Querydependentrankingusingk-nearestneighbor. [30] L.Vanderwende,H.Suzuki,C.Brockett,andA.Nenkova.
InSIGIR,2008. Beyondsumbasic: Task-focusedsummarizationwith
[15] A.Gionis,P.Indyk,R.Motwani,etal.Similaritysearchin sentencesimplificationandlexicalexpansion.Information
highdimensionsviahashing.InVLDB,volume99,1999. Processing & Management,43(6),2007.
[16] R.Gunning.Judgesscoldlawyersforbadwriting,1952. [31] L.Wang,H.Raghavan,V.Castelli,R.Florian,and
[17] J.Hoffart,S.Seufert,D.B.Nguyen,M.Theobald,and C.Cardie.Asentencecompressionbasedframeworkto
G.Weikum.Kore: keyphraseoverlaprelatednessforentity query-focusedmulti-documentsummarization.InACL,
disambiguation.InCIKM,2012. 2013.
[18] H.ImranandA.Sharan.Improvingeffectivenessofquery [32] S.Whiting,J.Jose,andO.Alonso.Wikipediaasatime
expansionusinginformationtheoreticapproach.InTrends machine.InWWW,pages857–862,2014.
in Applied Intelligent Systems.2010. [33] Z.WuandC.L.Giles.Measuringterminformativenessin
[19] T.Joachims.Optimizingsearchenginesusingclickthrough context.InNAACL-HLT,2013.
data.InKDD,2002. [34] R.Yan,X.Wan,J.Otterbacher,L.Kong,X.Li,and
[20] J.P.Kincaid,R.P.FishburneJr,R.L.Rogers,andB.S. Y.Zhang.Evolutionarytimelinesummarization: a
Chissom.Derivationofnewreadabilityformulas balancedoptimizationframeworkviaiterativesubstitution.
(automatedreadabilityindex,fogcountandfleschreading InSIGIR,2011.
easeformula)fornavyenlistedpersonnel.Technicalreport, [35] X.W.Zhao,Y.Guo,R.Yan,Y.He,andX.Li.Timeline
DTICDocument,1975. generationwithsocialattention.InSIGIR,2013.