Analysis and study on text representation to improve the accuracy of the Normalized Compression ...

202 Pages·2014·1.71 MB·English

Universidad Auto´noma de Madrid
Escuela Polit´ecnica Superior
2
1 ANALYSIS AND STUDY ON TEXT
0
2
REPRESENTATION TO IMPROVE

y
a
M THE ACCURACY OF THE

9 NORMALIZED COMPRESSION
2

DISTANCE
]
T
I
.
s
c
[

1
v
6
7
3 Thesis
6
.
5
0
2
1
:
v
i
X
r
a
Ana Granados Fontecha
Madrid, 2012
ADVISORS: David Camacho Fern´andez
Francisco de Borja Rodr´ıguez Ortiz

Tribunal nombrado por el Mgfco. y Excmo. Sr. Rector de la Universidad
Aut´onoma de Madrid, el d´ıa ....... de ................................ de 2012.
Presidente: D. ..........................................
Vocal: D. ..........................................
Vocal: D. ..........................................
Vocal: D. ..........................................
Secretario: D. ..........................................
RealizadoelactodedefensaylecturadelaTesiseld´ıa....... de..................
de 2012 en ..............................................................
Calificaci´on: ......................................................
EL PRESIDENTE EL SECRETARIO
LOS VOCALES

A mis padres.

After climbing a great hill, one only finds
that there are many more hills to climb.
-Nelson Mandela

Abstract
The huge amount of information stored in text form makes methods that
deal with texts really interesting. This thesis focuses on dealing with texts
using compression distances. More specifically, the thesis takes a small step
towards understanding both the nature of texts and the nature of compres-
sion distances. Broadly speaking, the way in which this is done is explor-
ing the effects that several distortion techniques have on one of the most
successful distances in the family of compression distances, the Normalized
Compression Distance -NCD-.
Theresearchcarriedoutinthisthesiscanbedividedintothreeparts. The
first part, which corresponds to Chapter 5, experimentally evaluates the im-
pact that several word removal techniques have on NCD-driven text cluster-
ing, with the aim of better understanding of both the nature of compression
distances and the nature of textual information. This goal is accomplished
by analyzing how the information contained in the documents and how the
upper bound estimation of their Kolmogorov complexity progress as words
are removed from the documents. One of the main conclusions that can be
drawn from this analysis is that the clustering accuracy can be improved
by applying a specific word removal technique. This distortion technique
consists of removing the most frequent words of the language preserving the
previous text structure.
The second part of the thesis, which corresponds to Chapter 6, attempts
toshedlightonthereasonswhytheapplicationofsuchadistortiontechnique
canimprove NCD-driven text clustering. The experimental results show that
the maintenance of boththe previous text structure and the remaining words
structure have some relevance in the clustering behavior.
The third part of the thesis, which corresponds to Chapter 7, applies the
above mentioned distortion technique to NCD-driven document search. The
application of compression distances to document search is not trivial due to
the fact that they do not commonly perform well when the compared objects
have very different sizes. An NCD-based document search engine that deals
with that drawback by using passage retrieval, is used in the third part of
the thesis. The results show that the search accuracy can be improved by
applying the distortion technique presented previously.
Summarizing, one of the distortion techniques explored in the thesis has
been found to be beneficial both in NCD-based document clustering and in
NCD-based document search.