Filtering Antonymous, Trend-Contrasting, and Polarity-Dissimilar

13 Pages·2011·0.19 MB·English

Filtering Antonymous, Trend-Contrasting, and Polarity-Dissimilar
Distributional Paraphrases for Improving Statistical Machine Translation
YuvalMarton AhmedElKholy and NizarHabash
T.J.WatsonResearchCenter CenterforComputationalLearningSystems
IBM ColumbiaUniversity
[email protected] {akholy,habash}@ccls.columbia.edu
Abstract 2008). It relies on parallel texts (or translation
phrase tables) in various languages, which are typ-
Paraphrases are useful for statistical machine ically scarce, and hence limit its applicability. Dis-
translation (SMT) and natural language pro- tributionalparaphrasing(Martonetal.,2009)gener-
cessing tasks. Distributional paraphrase gen- atesparaphrasesusingadistributionalsemanticdis-
eration is independent of parallel texts and
tance measure computed over a large monolingual
syntactic parses, and hence is suitable also
corpus.1 Monolingual corpora are relatively easy
forresource-poorlanguages,buttendstoerro-
andinexpensivetocollect,butdistributionalseman-
neouslyrankantonyms,trend-contrasting,and
polarity-dissimilar candidates as good para- ticdistancemeasuresareknowntorankantonymous
phrases. We present here a novel method and polarity-dissimilar phrasal candidates high. We
for improving distributional paraphrasing by therefore attempt to identify and filter out such ill-
filtering out such candidates. We evalu- suitedparaphrasecandidates.
ate it in simulated low and mid-resourced
A phrase pair may have a varying degree of
SMT tasks, translating from English to two
antonymy, beyond the better-known complete op-
quite different languages. We show statisti-
cally significant gains in English-to-Chinese posites (hot / cold) and contradictions (did / did
translation quality, up to 1 BLEU from non- not), e.g., weaker contrasts (hot / cool), contrast-
filtered paraphrase-augmented models (1.6 ing trends (covered / reduced coverage), or senti-
BLEU from baseline). We also show that ment polarity (happy / sad). Information extrac-
yieldinggainsintranslationtoArabic,amor-
tion, opinion mining and sentiment analysis litera-
phologically rich language, is not straightfor-
ture has been grappling with identifying such pairs
ward.
(Pang and Lee, 2008), e.g., in order to distinguish
positiveandnegativereviewsorcomments,ortode-
1 Introduction
tectcontradictions(Marneffeetal.,2008;Voorhees,
Paraphraserecognitionandgenerationhasproven 2008). We transfer some of the insights, data and
useful for various natural language processing techniquestotheareaofparaphrasingandSMT.We
(NLP) tasks, including statistical machine transla- distributionallyexpandasmallseedsetofantonyms
tion (SMT), information retrieval, query expansion, in an unsupervised manner, following Mohammad
documentsummarization,andnaturallanguagegen- et al. (2008). We then present a method for fil-
eration. We concentrate here on phrase-level (as tering antonymous and polarity-dissimilar distribu-
opposed to sentence-level) paraphrasing for SMT. tional paraphrases using the expanded antonymous
ParaphrasingisusefulforSMTasitincreasestrans- list and a list of negators (e.g., cannot) and trend-
lationcoverage–apersistentproblem,eveninlarge- decreasing words (reduced). We evaluate the im-
scalesystems. Twocommonapproachesare“pivot” pact of our approach in a SMT setting, where non-
and distributional paraphrasing. Pivot paraphrasing
translates phrases of interest to other languages and 1Other variants use a lexical resource in conjunction with
back (Callison-Burch et al., 2006; Callison-Burch, themonolingualcorpus(Mirkinetal.,2009;Marton,2010).
237
Proceedingsofthe6thWorkshoponStatisticalMachineTranslation,pages237–249,
Edinburgh,Scotland,UK,July30–31,2011.(cid:13)c2011AssociationforComputationalLinguistics
baselinetranslationmodelsareaugmentedwithdis- order. Forexample,ifphrisspokensoftly,then
tributional paraphrases. We show gains of up to spokenverysoftlywouldbefilteredout.
1 BLEU relative to non-filtered models (1.6 BLEU 7. Filter out every candidate cand that is antony-
from non-augmented baselines) in English-Chinese moustophr (SeeAlgorithm1below).
modelstrainedonsmallandmedium-largesizedata, 8. Output k-best remaining candidates above a
but lower to no gains in English-Arabic. The small certainsimilarityscorethresholdt.
trainingsizesimulatesresource-poorlanguages.
Most of the steps above are similar to, and have
The rest of this paper is organized as follows:
been elaborated in, Marton et al. (2009). Due to
We describe distributional paraphrase generation in
spacelimitations,weconcentrateonthemainnovel
Section 2, antonym discovery in Section 3, and
element here, which is the antonym filtering step,
paraphrase-augmented SMT in Section 4. We then
detailedbelow. Antonyms(largelyspeaking)areop-
reportexperimentalresultsinSection5,anddiscuss
posites,termsthatcontrastinmeaning,suchashot/
the implications in Section 6. We survey related
cold. Negatorsaretermssuchasnotandlost,which
work in Section 7, and conclude with future work
oftenflipthemeaningofthewordorphrasethatfol-
inSection8.
lows or contains them, e.g., confidence / lost confi-
2 DistributionalParaphrases dence. Details on obtaining their definitions and on
obtaining the antonymous pair list and the negator
Ourmethodimprovesonthemethodpresentedin
listaregiveninSection3.
Marton et al. (2009). Using a non-annotated mono-
lingual corpus, our method constructs distributional Algorithm1Antonymouscandidatefiltering
profiles (DP; a.k.a. context vectors) of focal words Givenanantonymouspairlist,anegatorlist,anda
or phrases. Each DPphr is a vector containing log- phrase-paraphrasecandidate(phr-cand)pairlist,
likelihood ratios of the focal phrase phr and each forallphr-candpairsdo
wordw inthecorpus. Givenaparaphrasecandidate forallwordsw inphr do
phrasecand,thesemanticdistancebetweenphrand ifwisalsoincand,andthereisanegatorup
cand is calculated using the cosine of their respec- totwowordsbeforeitineitherphr orcand
tiveDPs(McDonald,2000). FordetailsonDPsand (butnotboth!)then
distributionalmeasures,seeWeedsetal.(2004)and filteroutthispair
TurneyandPantel(2010). if w,ant is an antonymous pair, and ant is
The search of the corpus for paraphrase candi- in cand, and there is no negator up to two
datesisperformedinthefollowingmanner: words before w and ant, or there is such a
negatorbeforeboththen
1. For each focal phrase phr, build distributional
filteroutthispair
profileDP .
phr
2. Gather contexts: for each occurrence of phr,
keepsurrounding(leftandright)contextL R. 3 Antonyms,Trends,SentimentPolarity
3. For each such context, gather paraphrase can- Native speakers of a language are good at deter-
didates cand which occur between L and R in miningwhethertwowordsareantonyms(hot–cold,
other locations in the training corpus, i.e., all ascend–descend,friend–foe)ornot(penguin–clown,
candsuchthatL cand Roccurinthecorpus. cold–chilly, boat–rudder) (Cruse, 1986; Lehrer and
4. For each candidate cand, build a profile Lehrer, 1982; Deese, 1965). Strict antonyms apart,
DPcandandmeasureprofilesimilaritybetween therearealsomanywordpairsthatexhibitsomede-
DPcand andDPphr. greeofcontrastinmeaning,forexample,lukewarm–
5. Rank all cand according to the profile similar- cold, ascend–slip, and fan–enemy (Mohammad et
ityscore. al., 2008). Automatically identifying such con-
6. Filter out every candidate cand that textually trasting word pairs has many uses including detect-
entails phr: This is approximated by filtering ing and generating paraphrases (The lion caught
candifitswordsallappearinphr inthesame the gazel / The gazel could not escape the lion)
238
and detecting contradictions (Marneffe et al., 2008; ical relations have also received attention at the
Voorhees, 2008) (The inhabitants of Peru are well Educational Testing Services (ETS). They clas-
off / the inhabitants of Peru are poor). Of course, sify antonyms into contradictories (alive–dead,
such “contradictions” may be a result of differing masculine–feminine),contraries(old–young,happy-
sentiment, new information, non-coreferent men- sad), reverses (attack–defend, buy–sell), direction-
tions, or genuinely contradictory statements. Iden- als (front–back, left–right), incompatibles (happy–
tifying paraphrases and contradictions are in turn morbid, frank–hypocritical), asymmetric contraries
useful in effectively re-ranking target language hy- (hot–cool, dry–moist), pseudoantonyms (popular–
potheses in machine translation, and for re-ranking shy, right–bad), and defectives (default–payment,
queryresponsesininformationretrieval. Identifying limp–walk)(Bejaretal.,1991).
contrastingwordpairs(orshortphrasepairs)isalso As mentioned earlier, in addition to antonyms,
useful for detecting humor (Mihalcea and Strappar- there are other meaning-contrasting phenomena, or
ava, 2005), as satire and jokes tend to have contra- other ways to classify them, such as contrasting
dictionsandoxymorons. Lastly,itisusefultoknow trends and sentiment polarity. They all may have
which words contrast a focal word, even if only to varying degrees of contrast in meaning. Hereafter
filter them out. For example, in the automatic cre- wesometimebroadlyrefertoalloftheseasantony-
ation of a thesaurus it is necessary to distinguish mousphrases. Theantonymousphrasepairgenera-
near-synonymsfromcontrastingwordpairs. Distri- tionalgorithmthatweuseheredoesnotemployany
butionalsimilaritymeasurestypicallyfailtodoso. antonym-subclass-specifictechniques.
Instances of strong contrast are recorded to some
3.2 Detectingantonyms
extent in manually created dictionaries, but hun-
dredsofthousandsofothercontrastingpairsarenot. Mohammad et al. (2008) used a Roget-like the-
Further, antonyms can be of many kinds such as saurus, co-occurrence statistics, and a seed set of
those described in Section 3.1 below. We use the antonyms to identify the degree of antonymy be-
Mohammad et al. (2008) method to automatically tween two words, and generate a list of antony-
generatealargelistofcontrastingwordpairs,which mous words. The thesaurus divides the vocabulary
areusedtoidentifyfalseparaphrases. Theirmethod into about a thousand coarse categories. Each cat-
isbrieflydescribedinSection3.2. egory has, on average, about a hundred closely re-
lated words. (A word with more than one sense,
3.1 Kindsofantonyms
is listed in more than one category.) Mohammad
Antonyms can be classified into different kinds. et al. first determine pairs of thesaurus categories
Adetaileddescriptionofonesuchclassificationcan that are contrasting in meaning. A category pair
be found in Cruse (1986) (Chapters 9, 10, and 11), is said to be contrasting if it has a seed antonym
wheretheauthordescribescomplementaries(open– pair. A list of seed antonyms is compiled using 16
shut, dead–alive), gradable adjective pairs (long– affix patterns such as X and unX (clear–unclear)
short, slow–fast) (further classified into polar, over- and X and disX (honest–dishonest). Once a con-
lapping, and equipollent antonyms), directional op- trastingcategorypairisidentified,allthewordpairs
posites (up–down, north–south), (further classified acrossthetwocategoriesareconsideredtohavecon-
into antipodals, counterparts, and reversives), re- trasting meaning. The strength of co-occurrence
lational opposites (husband–wife, predator–prey), (as measured by pointwise mutual information) be-
indirect converses (give–receive, buy–pay), con- tween two contrasting word pairs is taken to be the
gruence variants (huge–little, doctor–patient), and degree of antonymy. This is based on the distri-
pseudo opposites (black–white). It should be butional hypothesis of antonyms, which states that
noted, however, that even though contrasting word antonymous pairs tend to co-occur in text more of-
pairs and antonyms have long been studied by ten than chance. Co-occurrence counts are made
linguists, lexicographers, and others, experts do from the British National Corpus (BNC) (Burnard,
not always agree on the scope of antonymy and 2000). The approach attains more than 80% accu-
the kinds of contrasting word pairs. Some lex- racyonGRE-styleclosestoppositequestions.
239
3.3 Detectingnegators f withsimilarityscoressim(f ,f),
i i
TheGeneralInquirer(GI)(Stoneetal.,1966)has
asim = asim +(1−asim )sim(f ,f) (2)
11,788 words labeled with 182 categories of word i i−1 i−1 i
tags, such as positive and negative semantic orien-
where asim = 0. We only augment the phrase
tation, pleasure, pain, and so on.2 Two of the GI 0
tablewithasinglerulefromf toe,andinitarethe
categories, NOTLW and DECREAS, contain terms
featurevaluesofthephrasef forwhichsim(f ,f)
i i
that negate the meaning of what follows (Choi and
wasthehighest.
Cardie, 2008; Kennedy and Inkpen, 2005). These
terms (with limited added inflection variation) form 5 Experiment
ourlistofnegators.
5.1 SystemandParameters
4 Paraphrase-AugmentedSMT We augmented translation models with para-
phrases based on distributional semantic distance
AugmentingthesourcesideofSMTphrasetables
measures, with our novel antonym-filtering, and
withparaphrasesofout-of-vocabulary(OOV)items
without it. We tested all models in English-
was introduced by Callison-Burch et al. (2006),
to-Chinese and English-to-Arabic translation, aug-
and was adopted practically ‘as-is’ in consequent
menting the models with translation rules for un-
work (Callison-Burch, 2008; Marton et al., 2009;
known English phrases. We also contrasted these
Marton, 2010). Given an OOV source-side phrase
modelswithnon-augmentedbaselinemodels.
f, if the translation model has a rule hf0,ei whose
For baseline we used the phrase-based SMT sys-
source side is a paraphrase f0 of f, then a new rule
tem Moses (Koehn et al., 2007), with the default
hf,ei is added, with an extra weighted log-linear
model features: 1. phrase translation probability,
feature, whose value for the new rule is the similar-
2. reverse phrase translation probability, 3. lexical
ity score between f and f0 (computed as a function
translation probability, 4. reverse lexical translation
of the pivot translation probabilities or the distribu-
probability,5.wordpenalty,6.phrasepenalty,7.six
tional semantic distance of the respective DPs). We
lexicalized reordering features, 8. distortion cost,
followthesamelinehere:
and 9. language model (LM) probability. We used
Giza++ (Och and Ney, 2000) for word alignment.
 All features were weighted in a log-linear frame-
asimD(DPfP)f0, Iifspghernaesreattaebdlefreonmtr(ye(,ef,0f)) wseotrwkit(hOmchinaimndumNeeyr,ro2r0r0a2te).traFienaitnugre(Owcehi,g2h0ts03w)eorne
h(e,f)= usingmonolingually-
1 Odtehreivrewdispea.raphrases. oabtujencitnigvesefutnucsitniognB. TLeEsUtr(ePsauplitnsewneireetaelv.a,l2u0a0te2d)aussitnhge
BLEU and TER (Snover et al., 2006): The higher
(1)
the BLEU score, the better the result; the lower the
TER score, the better the result. This is denoted
where the definition of asim is repeated below. As
withBLEU↑andTER↓inTable1. Statisticalsignif-
noted in that previous work, it is possible to con-
icance of model output differences was determined
struct a new translation rule from f to e via more
using Koehn (2004)’s test on the objective function
than one pair of source-side phrase and its para-
(BLEU).
phrase; e.g., if f is a paraphrase of f, and so is f ,
1 2
The paraphrase-augmented models were created
andbothf ,f translatetothesamee,thenbothlead
1 2
as described in Section 4. We used the same data
totheconstructionofthenewruletranslatingf toe,
and parameter settings as in Marton (2010).3 We
butwithpotentiallydifferentfeaturescores. Inorder
used cosine distance over DPs of log-likelihood ra-
to leverage on these paths and resolve feature value
tios (McDonald, 2000), built with a sliding win-
conflicts, an aggregated similarity measure was ap-
plied: Foreachparaphrasef ofsource-sidephrases 3Data preprocessing and paraphrasing code slightly differ
fromthoseusedinMartonetal.(2009)andMarton(2010),and
2http://www.wjh.harvard.edu/∼inquirer hencescoresarenotexactlythesameacrossthesepublications.
240
dow of size ±6, a sampling threshold of 10000 oc- NISTMT2005evaluationset. Inordertouseitfor
currences, and a maximal paraphrase length of 6 the reverse translation direction (English-Chinese),
tokens. We applied a paraphrase score threshold we arbitrarily chose the first English reference set
t = 0.05; a dynamic context length (the short- as the tuning “source”, and the Chinese source as a
est non-stoplisted left context L occurring less than single “reference translation”. For testing we used
512 times in the corpus, and similarly for R); para- the English-Chinese NIST MT evaluation 2008 test
phrasingofOOVunigrams;filteringparaphrasecan- setwithitsfourreferencetranslations.
didates occurring less than 25 times in the corpus
English-Arabic: We use an English-Arabic par-
(inspired by McDonald, 2000); and allowing up to
allel corpus of about 135k sentences (4 million
k = 100 best paraphrases per phrase. We tuned
words) and a subset of 30K sentences (one mil-
the weights of each model (non-augmented base-
lionwords)forthetranslationmodels’trainingdata.
line, unigram-augmented, and unigram-augmented-
The sentences were extracted from Arabic News
filtered)withaseparateminimumerrorratetraining.
(LDC2004T17), eTIRR (LDC2004E72), English
We explored here augmenting OOV unigrams, translation of Arabic Treebank (LDC2005E46),
although our paraphrasing and antonym filtering and Ummah (LDC2004T18).4 For Arabic pre-
methods can be applied to longer n-grams with no processing, we follow previously reported best to-
furthermodifications. However,preliminaryexperi- kenization scheme (TB)5 and orthographic word
mentsshowedthatlongern-gramsrequireadditional normalization condition (Reduced) when translat-
provisionsinordertoyieldgains. ing from English to Arabic (El Kholy and Habash,
5.2 Data 2010b). MADA (Habash and Rambow, 2005) is
usedtopre-processtheArabictextforthetranslation
InordertotakeadvantageoftheEnglishantonym
modeland5-gramlanguagemodel(LM).Asapost-
resource, we chose English as the source language
processing step, we jointly denormalize and deto-
for the translation task. We chose Chinese as
kenize the text to produce the final Arabic output.
the translation target language in order to compare
Following El Kholy and Habash (2010a), we use
withMarton(2010),andforthesamereasonsitwas
their best detokenization technique, T+R+LM. The
chosenthere: ItisquitedifferentfromEnglish(e.g.,
techniquecruciallyutilizesalookuptable(T),map-
in word order), and four reference translation were
ping tokenized forms to detokenized forms, based
available from NIST. We chose Arabic as another
onourMADA-fiedLM.Alternativesaregivencon-
target language, because it is different from both
ditional probabilities, P(detokenized|tokenized).
English and Chinese, and richer morphologically,
Tokenized words absent from the tables are deto-
whichintroducesadditionalchallenges.
kenized using deterministic rules (R), as a backoff
English-Chinese: For training we used the
strategy. We use a 5-gram untokenized LM and
LDC Sinorama and FBIS tests (LDC2005T10 and
the disambig utility in the SRILM toolkit to de-
LDC2003E14), and segmented the Chinese side
cide among different alternatives. Word alignment
with the Stanford Segmenter (Tseng et al., 2005).
is done using GIZA++, as in English-Chinese sys-
Aftertokenizationandfiltering,thisbitextcontained
tem. We use lemma-based alignment which consis-
231,586 lines (6.4M + 5.1M tokens). We trained a
tentlyyieldssuperiorresultstosurface-basedalign-
trigram language model on the Chinese side, with
ment (El Kholy and Habash, 2010b). For LM, we
the SRILM toolkit (Stolcke, 2002), using the mod-
use200MwordsfromtheArabicGigawordCorpus
ified Kneser-Ney smoothing option. We followed
(LDC2007T40)togetherwiththeArabicsideofour
the split in Marton (2010), and constructed the re-
trainingdata.
duced set of about 29,000 sentence pairs. The pur-
All experiments were conducted using Moses
poseofcreatingthissubsetmodelwastosimulatea
here as well. We used a maximum phrase length
resource-poor language. We trained separate trans-
lationmodels,usingeitherthesubsetorthefull-size
4All are available from the Linguistic Data Consortium
trainingdataset.
(LDC)http://www.ldc.upenn.edu
For weight tuning we used the Chinese-English 5TB:PennArabicTreeBanktokenizationscheme
241
of size 8 tokens. Weight optimization was done us- only one entry was antonym-filtered here, we do
ing a set of 300 sentences from the NIST MT 2004 not provide separate scores for aug-1gram-ant-filt.
Arabic-Englishevaluationtestset(MT04). Thetun- Surprisingly, for the reduced 30k models, all scores
ing was based on tokenized Arabic without detok- (BLEU,TER,andeventheirlemmatizedvariants)of
enization. Testing was done on the NIST Arabic- the augmented 1gram model were somewhat worse
English MT05 and MEDAR 2010 English-Arabic thanthebaseline’s,andthoseoftheantonym-filtered
four-reference evaluation sets. For both tuning on modelweretheworst. wealsorana4-referencetest
MT04 and testing on MT05, since we need the re- (Medar) to see whether the single MT05 reference
verse English-Arabic direction, we chose one En- wasproblematic,butresultsweresimilar. Weexam-
glish reference translation as the “source”, and the inepossiblereasonsforthisinthenextsection.
Arabic as a single “reference”. We evaluated using
BLEUandTERheretoo. 6 Discussion
English paraphrases: We augmented the base-
Filtering quality: Our filtering technique is based
linemodelswithparaphrasesgeneratedasdescribed
on antonymous pair and negator lists that were ex-
above, using a monolingual text of over 516M to-
panded distributionally from seed sets. Therefore,
kens, consisting of the BNC and English Gigaword
they are noisy. From a small random sample (Ta-
documents from 2004 and 2008 (LDC2009T13),
ble3)itseemsthatonlyabout10%offilteredcases
pre-processedtoremovepunctuationandtoconflate
shouldnothavebeenfiltered;oftherest, 50%were
numbers,dates,months,daysofweek,andalphanu-
stronglyantonymous,25%mildlyso,and15%were
merictokenstotheirrespectiveclasses.
siblings (co-hypernyms) in a natural categorical hi-
5.3 Results erarchy or otherwise noisy paraphrases filtered due
to a noisy antonym pair. Negators in the unigrams’
English-Chinese: Results are given in Table 1.
paraphrasecandidateswererare.
AugmentingSMTphrasetableswithparaphrasesof
OOV unigrams resulted in gains of 0.6-0.7 BLEU English-Chinese: Our paraphrase filtering tech-
points for both subset and full models, but TER nique yielded an additional 1 BLEU point gain
scoreswereworse(higher)forthefullmodel. Aug- overthenon-filteredparaphrase-augmentedreduced
mentingsamemodelswithsameparaphrasesfiltered model (totaling 1.6 BLEU over baseline). The re-
for antonyms resulted in further gains of 1.6 and 1 duced and large augmented models’ phrase table
BLEUpointsforbothsubsetandfullmodels,respec- sizeincreasedbyabout27%and4%,respectively–
tively, relativeto therespectivebaselines. TheTER andantonymfilteringdidnotchangethesenumbers
scores of the antonym filtered models were also as bymuch(seeleftsideofTable4). Therefore,thedif-
goodorbetter(lower)thanthoseofthebaselines. ferenceinperformancebetweenthefilteredandnon-
filteredsystemsisunlikelytobequantitative(phrase
reducedsize largesize tablesize). Theoutofvocabulary(OOV)rateofthe
model BLEU↑ TER↓ BLEU↑ TER↓ 29k subset model is somewhat high (see Table 4),
especiallyforthetestset;butonlyaftertheseexper-
baseline 15.8 69.2 21.8 63.8
aug-1gram 16.4B 68.9 22.5B 64.4 iments were completed did we peek at the test set
aug-1gram-ant-filt 17.4BD 68.7 22.8BD 63.7 for calculating these statistics, and in any case, we
shouldnotbeguidedbysuchinformationinchoos-
Table1:English-Chinesescores.B/D=statisticallysignificant
ing the test set. At first glance it may seem surpris-
w.r.t. (B)aselineor(D)istributional1grammodel,usingKoehn
ing that only 0.4% of the paraphrase candidates of
(2004)’sstatisticalsignificance.
the English OOV unigrams (248 candidates) were
English-Arabic: Results are given in columns 1-7 filtered by our procedure, and that it accounted for
ofTable2. OntheMT05testset,the135k-sentence asmuchas1BLEUinthereducedset. (ForEnglish-
aug-1gram model outperformed its baseline in both Arabic only 0.6%, or 23 candidates, were filtered).
BLEU and TER scores. The lemmatized variants Leavingtheestimationofantonymousphrasedetec-
of the scores showed higher or same gains. Since tion recall for the future, we note that these num-
242
BLEU Lemm. Brev. Ref/Sys TER Lemm. UnigramLemmaMatchAnalysis
↑ BLEU penal. ratio ↓ TER ExactMatch Lemma-only Unmatchable Total
30k-sentence(1Mword)trainingdatasetmodels
MT05baseline 23.6 31.3 99.2 1.008 57.6 47.3 15614 55.4% 4055 14.4% 8550 30.3% 28219
aug-1gram 23.2 30.8 99.9 1.001 58.8 48.4 15387 54.2% 4195 14.8% 8831 31.1% 28413
aug-1gram-ant-filt 23.2 30.8 99.9 1.001 58.8 48.3 15387 54.2% 4195 14.8% 8831 31.1% 28413
MEDARbaseline 13.6 18.7 93.6 1.066 67.6 61.3 4924 53.0% 1563 16.8% 2800 30.1% 9287
aug-1gram 12.9 18.3 94.2 1.060 68.9 62.3 4894 52.0% 1710 18.2% 2815 29.9% 9419
aug-1gram-ant-filt 12.9 18.3 94.2 1.060 69.0 62.3 4891 51.9% 1715 18.2% 2815 29.9% 9421
135k-sentence(4Mword)trainingdatasetmodels
MT05baseline 25.8 33.5 99.2 1.008 55.7 45.3 16115 57.1% 3999 14.2% 8128 28.8% 28242
aug-1gram 26.4 34.3B 99.5 1.005 55.1 44.7 16156 57.1% 4068 14.4% 8089 28.6% 28313
aug-1gram-ant-filt 26.4 34.3B 99.5 1.005 55.0 44.6 16153 57.1% 4090 14.5% 8068 28.5% 28311
MEDARbaseline 17.1 23.1 94.7 1.054 65.1 58.6 5483 57.7% 1577 16.6% 2438 25.7% 9498
aug-1gram 17.2 23.5 95.3 1.048 65.1 58.6 5586 58.1% 1606 16.7% 2424 25.2% 9616
aug-1gram-ant-filt 17.2 23.5 95.3 1.048 65.1 58.6 5586 58.1% 1606 16.7% 2424 25.2% 9616
Table2: English-ArabictranslationscoresandanalysisforNISTMT05andMEDARtestsets. B=statisticallysignificantw.r.t.
(B)aselineusingKoehn(2004)’sstatisticalsignificancetest.
bersfromEnglisharenotdirectlycomparabletothe Whileindividualunigramto4gramscoresforthe
Chinese side: they relate to paraphrase candidates augmented models were lower than the baseline’s,
andnotphrasetableentries;theyrelatetotypesand filtered model’s unigram and bigram scores were
not tokens; each OOV English word may translate lower or similar to the baseline’s, and their trigram
to one or more Chinese words, each of which may and 4gram scores were higher than the baseline’s.
compriseofoneormorecharacters;andlastbutnot We intend to further investigate the cause for this
least,the BLEUscoreweuseischaracter-based. pattern,anditseffectontranslationquality,withthe
helpofanativeChinesespeaker–andonBLEU,to-
phrase |||paraphrase |||score comments getherwiththebrevitypenalty–inthefuture.
absence |||occupation |||0.06 mild English-Arabic: Themoststrikingfactisthesetof
absence |||presence |||0.33 good differences between the language pairs: In English-
backwards |||forwards |||0.21 good Chinese,weseegainswithdistributionalparaphrase
wooden |||plasticlawn |||0.12 sibling
augmentation, and further gains when antonymous
dump |||disposeof |||0.41 bad
cooler |||warm |||0.45 mild and contrasting paraphrase candidates are filtered
diminished |||increased |||0.23 good out. Butinthe30k-sentenceEnglish-Arabicmodels,
minor |||serious |||0.42 good paraphrase augmentation actually degrades perfor-
relic |||youthactivistinthe |||0.12 harmless
mance, even in lemma scores. It has been observed
dive |||rise |||0.15 good
argue |||alsorecognize |||0.05 mild before that BLEU (and similarly TER) is not ideal
bother |||wastetime |||0.79 bad forevaluationofcontributionsofthissort(Callison-
dive |||climb |||0.17 good Burch et al., 2006). Therefore we conducted both
moonlight |||spring |||0.05 harmless
manual and focused automatic analysis, including
sharply |||slightly |||0.60 good
OOV statistics and unigram lemma match analysis6
substantial |||meager |||0.14 good
warmer |||cooler |||0.72 good
tough |||delicate |||0.07 good 6Unigramlemmamatchanalysisisaclassificationofallthe
tiny |||mostlymuslim |||0.06 mild
wordsinthetranslationhypothesis(againstthetranslationref-
softly |||deep |||0.06 mild
erence)into:(a)exactmatch,whichisequaltosimpleunigram
precision,(b)lemma-onlymatch,whichcountswordsthatcan
Table3: Randomfilteringexamples onlybematchedatthelemmalevel,and(c)unmatchable.
243
between the system output and the reference trans- ErrorAnalysis Weconductedanerroranalysisof
lation. our Arabic 30k system using part of the MT05 test
Table 4 shows that the OOV rates for English- set. That set had 571 OOV types, out of which,
Arabic are lower than English-Chinese. But if they we were able to augment phrases for 196 OOV
were negligible, we would not expect to see gains types. The majority of OOV words were proper
(orinfactanychange)ineithermodelsize,contrary nouns(67.8%),withtherestbeingmostlynouns,ad-
to fact. It is interesting to point out that our trans- jectives and verbs (in the order of their frequency).
lation model augmentation technique handles about AmongtheOOVsforwhichweaugmentedphrases,
50%ofthe(non-digit,non-punctuation)OOVwords the proper noun ratio was smaller than the full set
in all models (except for only half that in the 135k (45.4% relative). We selected a random sample of
model,whichstillshowedgains). 50 OOV words, and examined their translations in
the MT05 test set. The analysis considered all the
Anotherconcernisthatthecurrentmaximalpara-
OOV word occurrences (96 sentences). We classi-
phrase length (6 tokens) may be too far from the
fied each OOV translation in the augmented system
paraphrasee’s length (unigram), resulting in lower
andtheaugmented-filteredsystemasfollows:
quality. However, a closer examination of the
length difference evident through the BLEU brevity a1 correct(andinreference)
penalty and the reference:system-output length ra- a2 correct(morphologicalvariation)
a3 acceptabletranslationintoasynonym
tio (columns 4-5 of Table 2), reveals that the dif-
a4 acceptabletranslationintoahypernym
ferences are small and inconsistent; on average, the
b1 wrongtranslationintoahypernym
brevity penalty difference accounts for roughly 0.1
b2 co-hypernym: a sibling in a psychologically
absolute BLEU points and 0.2 absolute lemmatized
naturalcategoryhierarchy
BLEUpointsoftherespectivedifferences.7 b3 antonymous, trend-contrasting, or polarity dis-
Last, Modern Standard Arabic is a morphologi- similarmeaning
cally rich language: It has many inflected forms for c1 wrongproper-nountranslation(sibling)
most verbs, and several inflected forms for nouns, c2 wrongproper-nountranslation(other)
adjectives and other parts of speech – and complex d wrongtranslationforotherreasons
syntactic agreement patterns showing these inflec-
Both the augmented and augmented-filtered system
tions. It might be the case that the inflected Arabic
had 27.1% correct cases (category a). Only one-
LMmodelmightnotservewelltheaugmentedmod-
quarter of these were exact matches with the refer-
els,sincetheyincludetranslationrulesthataremore
ence (category a1) that can be captured by BLEU.
likely to be “off” inflection-wise (e.g., showing un-
Incorrect proper-noun translation (category c) was
grammatical syntactic agreement or simply an ac-
thebiggesterror(augmentedmodel: 33.3%,filtered
ceptablechoicethatdiffersfromthereference). Pre-
model: 37.5%); within this category, sibling mis-
sumably, the smaller the training set, the larger this
translations (category c1), e.g., Buddhism is trans-
problem,sincetherewouldbefewerrulesandhence
lated as Islam, were the majority (over half in aug-
smaller variety of inflected forms per similar core
mented model, and about two-thirds in the filtered
meaning. The unigram lemma match analysis and
model). Proper nouns seem to be a much bigger
lemma scores’ statistics (Table 2) support this con-
problem for translation into Arabic than into Chi-
cern. In the 30k model, lemma-only match seems
nese in our sets. Category b mis-translations ap-
to even further increase, at the expense of the exact
peared in 20.8% of the time (equally in augmented
word-form match. Possible solutions include using
andfiltered). Almosthalfoftheseweresiblingmis-
a lemma-based LM, or another LM that is adjusted
translations (category b2), e.g., diamond translated
tothissortofinflection-wise“off”text.
as gold. Only two OOV translations in our sam-
ple were antonymous (category b3). It is possible,
7These values are computed by subtracting the difference
therefore, that our Arabic sets do not give room for
betweentwoBLEUscoresfromthedifferencebetweenthesame
twoBLEUscoreswithouttheeffectofbrevitypenalty(i.e.,each our filtering method to be effective. (cid:135)I(cid:16)n(cid:210)(cid:170)o(cid:16)(cid:75)ne case,
dividedbyitsbrevitypenalty). the verbdeepen(reference translation ) ismis-
244
(cid:16) (cid:16)
(cid:233)(cid:212)(cid:175)
translated as summit ( ). In the other case, the usingadirectChinese-Arabicbilingualparalleltext.
adjective cool (political relations), whose reference OtherattemptstoreducetheOOVratebyaugment-
t(cid:81)ra(cid:16)(cid:75)(cid:241)n(cid:16)(cid:74)s(cid:203)l(cid:64)at(cid:225)i(cid:9)o(cid:211)n(cid:72)u(cid:16)s(cid:64)e(cid:81)s(cid:16)(cid:30)(cid:175)(cid:9)afigureofspeechperiodsofte(cid:16)(cid:232)n(cid:89)s(cid:74)(cid:10)i(cid:107)o(cid:46)n ing the phrase table’s source side include Habash
( ), is mistranslated as good ( ), (2009), providing an online tool for paraphrasing
which carries the opposite sentiment. The rest of OOV phrases by lexical and morphological expan-
category b involve hypernyms (b1), such as(cid:16)tra(cid:17)ns- sion of known phrases and dictionary terms – and
(cid:233)(cid:187)(cid:81)(cid:229)(cid:132)(cid:203)(cid:64)
latingtheOOVwordtelecomintocompany( ). transliterationofpropernames.
Overall,thefilteredmodeldidnotbehavesignifi- Bond et al. (2008) also pivot for paraphrasing.
cantlydifferentlyfromitsaugmentedcounterpart. They improve SMT coverage by using a manually
crafted monolingual HPSG grammar for generating
Chinese-Arabic score difference: We conjecture
meaningandgrammar-preservingparaphrases. This
that another possible reason for the different score
grammarallowsforcertainwordreordering,lexical
gain patterns between the two language pairs is the
substitutions,contractions,and“typo”corrections.
factthatinChinese,manywordsthataresiblings-in-
meaning share a character, which doesn’t necessar- Onishi et al. (2010), Du et al. (2010), and others,
ilyhaveastand-alonemeaning;therefore,character- pivot-paraphrase the input, and represent the para-
based BLEU was able to give credit to such para- phrasesinalatticeformat,decodingitwithMoses.
phrasesontheChineseside,whichwasnotcasefor Work on paraphrase generation: Barzilay and
theword-based BLEUontheArabicside. McKeown(2001)extractparaphrasesfromamono-
lingual parallel corpus, containing multiple transla-
7 RelatedWork
tions of the same source. However, monolingual
parallelcorporaareextremelyrareandsmall. Dolan
This paper brings together several sub-areas:
et al. (2004) use edit distance for paraphrasing.
SMT, paraphrase generation, distributional seman-
Max (2009) and others take the context of the para-
tic distance measures, and antonym-related work.
phrasedword’soccurrenceintoaccount. Zhaoetal.
Therefore we can only briefly survey the most rel-
(2008) apply SMT-style decoding for paraphrasing,
evantworkhere. Ourworkcanbeviewedasanex-
using several log linear weighted resources while
tensionofthelineofresearchthatseekstoaugment
Zhao et al. (2009) filter out paraphrase candidates
translationtableswithautomaticallygeneratedpara-
andweightparaphrasefeaturesaccordingtothede-
phrases of OOV words or phrases in a fashion sim-
sired NLP task. Chevelu et al. (2009) introduce
ilar to Section 4: Callison-Burch et al. (2006) use
a new paraphrase generation tool based on Monte-
pivoting technique (translating to other languages
Carlo sampling. Mirkin et al. (2009), inter alia,
and back) in order to generate paraphrases, and the
frameparaphrasingasaspecial,symmetricalcaseof
pivottranslationprobabilityastheirsimilarityscore;
(WordNet-based) textual entailment. See Madnani
Callison-Burch(2008)filterssuchparaphrasesusing
andDorr(2010)foragoodparaphrasingsurvey.
syntactic parsing information; Marton et al. (2009)
use distributional paraphrasing technique that ap- Work on measuring distributional semantic dis-
plies distributional semantic distance measure for tance: For one survey of this rich topic, see Weeds
the paraphrase score; Marton (2010) applies a lexi- et al. (2004) and Turney and Pantel (2010). We
calresource/corpus-basedhybridsemanticdistance useherecosineoflog-likelihoodratios(McDonald,
measure for the paraphrase score instead, approxi- 2000). A recent paper (Kazama et al., 2010) advo-
mating word senses; here, we apply a distributional cates a Bayesian approach, making rare terms have
semantic distance measure that is similar to Marton lowerstrengthofassociation, asaby-productofre-
et al. (2009), with the main difference being the fil- lyingontheirprobabilisticExpectation.
teringoftheresultingparaphrasesforantonymity.
Work on detecting antonyms: Our work with
Other work on augmentating SMT: Habash and antonymscanbethoughtofasanapplication-based
Hu (2009) show, pivoting via a trilingual parallel extension of the (Mohammad et al., 2008) method.
text, that using English as a pivot language be- Some of the earliest computational work in this
tween Chinese and Arabic outperforms translation area is by Lin et al. (2003) who used patterns
245
model e2z:29k e2z:232k e2a:30k e2a:135k
phrasetablebaselinevocab.(#source-sidetypes) 13916 34825 24371 49854
phrasetableentries:baseline 1996k 13045k 2606k 12344k
phrasetableentries:aug-1gram 2543k 127.38% 13615k 104.37% 2635k 101.09% *12373k 100.23%
phrasetableentries:aug-1gram-ant-filt 2542k 127.35% 13615k 104.37% 2635k 101.09% *12373k 100.23%
OOVtypesintune(%tunetypes) 1097 21.58% 451 8.87% 141 7.31% 84 4.35%
OOVtokensintune(%tunetokens) 2138 6.10% 917 2.62% 193 2.18% 115 1.30%
OOVtypesintest(%testtypes) 2473 33.59% 1227 16.66% 574 12.42% 339 7.34%
OOVtokensintest(%testtokens) 4844 10.40% 2075 4.46% 992 2.83% 544 1.55%
tuneOOVtokendecreaseinaug-1gram/ant-filt 1343 27.73% 510 24.58% 79 7.96% 28 5.15%
tuneOOVtypedecreaseinaug-1gram/ant-filt 646 58.89% 203 45.01% 60 42.55% 22 26.19%
testOOVtokendecreaseinaug-1gram/ant-filt 2776 57.31% 996 48.00% 460 46.37% 127 23.35%
testOOVtypedecreaseinaug-1gram/ant-filt 1394 56.37% 585 47.68% 246 42.86% 76 22.42%
Table4:Out-of-vocabulary(OOV)wordratesandphrasetablesizesforallmodelsizesandlanguagepairs.e2z=English-Chinese;
e2a=English-Arabic.Thestatisticsmarkedwith*inthetop-rightcellareidentical,see§5.3.
such as “from X to Y” and “either X or Y” to We also showed for the first time transla-
distinguish between antonymous and similar word tion gains for English-Arabic with paraphrase-
pairs. Harabagiu et al. (2006) detected antonyms augmented (non-filtered) models. However, Ara-
by determining if their WordNet synsets are con- bic,andpresumablyothermorphologicallyrichlan-
nected by the hypernymy–hyponymy links and ex- guages, may require more complex models in order
actlyoneantonymylink. Turney(2008)proposeda tobenefitfromourfilteringmethod.
supervised method to solve word analogy questions Our antonym detection and filtering method is
thatrequireidentifyingsynonyms,antonyms,hyper- distributional and heuristic-based; hence it is noisy.
nyms, and other lexical-semantic relations between We suspect that OOV terms in larger models tend
wordpairs. to be harder to paraphrase (judging by the differ-
ence from the reduced models, and the lower OOV
8 ConclusionsandFutureWork rate),andalsohardertofilterparaphrasecandidates
of(duetothelowerparaphrasequality,whichmight
Wepresentedhereanovelmethodforfilteringout
not even include sufficiently distributionally similar
antonymous phrasal paraphrase candidates, adapted
candidates,antonymousorotherwise). Inthefuture,
fromsentimentanalysisliterature,andtestedinsim-
we intend to improve our method, so that it can be
ulatedlow-andmid-resourcedSMTtasksfromEn-
used to improve also the quality of models trained
glish to two quite different languages. We used an
onevenlargerparalleltexts.
antonymouswordpairlistextracteddistributionally
Last,weintendtoextendourmethodbeyondun-
byextendingaseedlist. Then,theextendedlist,to-
igrams,limitparaphraselengthtothevicinityofthe
getherwithanegatorlistandanovelheuristic,were
paraphrasee’slength,andimproveourinflectedAra-
usedtofilteroutantonymousparaphrasecandidates.
bicgenerationtechnique,soitcanhandlethisnovel
Finally, SMT models were augmented with the fil-
typeofaugmenteddatawell.
teredparaphrases,yieldingEnglish-Chinesetransla-
Acknowledgments
tionimprovementsofupto1 BLEU fromthecorre-
sponding non-filtered paraphrase-augmented model
Part of this work was done while the first author
(up to 1.6 BLEU from the corresponding baseline wasatColumbiaUniversity. Thesecondauthorwas
model). Our method proved effective for mod-
funded through a Google research award. The au-
els trained on both reduced and mid-large English-
thors wish to thank Saif Mohammad for providing
Chinese parallel texts. The reduced models sim-
hisdataandforusefuldiscussion,andalsothankthe
ulated “low density” languages by limiting the
anonymousreviewersfortheirusefulfeedback.
amountofthetrainingtext.
246