Using data-compressors for statistical analysis of problems on homogeneity testing and classification

0.22 MB·

by  Boris Ryabko

Using data-compressors for statistical analysis
of problems on homogeneity testing and
classification
Boris Ryabko Andrey Guskov
Institute of Computational Technologies of SB RAS The State Public Scientific Technological
Novosibirsk, Russian Federation Library of SB RAS, Novosibirsk, Russian Federation
7 Irina Selivanova
1
The State Public Scientific Technological
0
Library of SB RAS, Novosibirsk, Russian Federation
2

n
a
J Abstract—Nowadays data compressors are applied to compresses sequences x1x2...xn, y1 y2 ... yk and com-
5 manyproblemsoftextanalysis,butmanysuchapplications binedonesx x ...x z z ...z andy y ...y z z ...z ,
1 2 n 1 2 m 1 2 k 1 2 m
1 are developed outside of the framework of mathematical the difference |ϕ(x x ...x z z ...z )|−|ϕ(x x ...x )|
1 2 n 1 2 m 1 2 n
statistics.Inthispaperweovercomethisobstacleandshow
willbelessthan|ϕ(y y ...y z z ...z )|−|ϕ(y y ...y )|
] howseveralmethodsofclassicalmathematicalstatisticscan 1 2 k 1 2 m 1 2 k
T bedevelopedbasedonapplicationsofthedatacompressors. (Here |U| is the length U.) For instance, let x1x2...xn,
I z z ...z be texts in English, whereas y y ... y is
1 2 m 1 2 k
.
s keywords: data compression, hypothesis testing, ho- in German. Then the English text z z ...z will be
1 2 m
c
mogeneity test, classification, universal code. compressed better after the text in the same language
[
(x x ...x ) than after the text in German (y y ...y ),
1 2 n 1 2 k
1 I. INTRODUCTION i.e. the first difference will be less than the second one.
v
Data compression methods (or universal codes) were This natural approach was used for diagnostic of the
8
discovered in the 1960’s and nowadays they are widely authorship of literary and musical texts, for estimation
2
0 used to compress texts for their storage or transmission. of closeness of DNA sequences, construction of phy-
4 In the last thirty years, it was recognized that data logenetic trees and many other problems ( [3], [4],
0 compressors can be used for many purposes which are [6], [10], [11], [18], [19]). Many papers (see [11] and
.
1 farfromfilecompaction.Inparticular,itwasshownthat review there) were devoted to the measurement of the
0 methods of data compression can be used for prediction interdependence between sequences (or the association,
7
andhypothesistestingfortimeseriesintheframeworkof similarity, closeness, etc.), because such measures plays
1
classical mathematical statistics, see [15] and a review an important rule in clustering, classification and some
:
v there. Later, several authors applied data compressors other methods of text analysis. It is important to note,
i
X to problems which are close, in spirit, to homogene- that their approaches are outside of the framework of
ity testing, estimation of correlation and covariance, mathematical statistics and, in particular, do not give a
r
a classifications, clustering and some others; see [3], [4], possibilitytoreasonaboutconsistencyofestimates,tests,
[10], [11], [18]. The main idea of their approach can classifiers, clustering, etc.
be understood from the following example. Suppose It is important that the modern data compressors
that there are three sequences of letters x x ...x , y are based on so-called universal codes and the main
1 2 n 1
y ... y , z z ...z and a certain data-compressor ϕ. properties of universal codes are valid for them (as far
2 k 1 2 m
The sequences x x ...x , y y ... y obey different asasymptoticpropertiescanbevalidforarealcomputer
1 2 n 1 2 k
probabilitydistributions,whereasz z ...z obeysoneof program). A formal definition of universal codes is
1 2 m
them.Thegoalistodeterminethisdistribution.(Itisthe given in Appendix 1, but here we informally note that
well-known “three samples problem”.) If x x ...x and universal codes can compress sequences generated by a
1 2 n
z z ...z obey the same probability distribution, then, source with unknown statistics till its Shannon entropy,
1 2 m
the sequence z z ...z will be compressed better after which, in turn, is a lower limit on lossless compression.
1 2 m
x x ...x than after y ... y . More precisely, if one Note that nowadays there are many classes of universal
1 2 n 1 k
codes which are based on different ideas and showed III. THEMAINTHEOREMS
their practical efficiency. Out of the most popular we
First we present a theorem which can be considered
mention the PPM algorithm, which is used along with
asatheoreticalbasisforapplicationofdatacompressors
thearithmeticcode([1],[14]),Lempel-Zivcodes([20],
for solving the problems described above. First, for two
[21]), Borrows-Willer transformation ( [2], [12]) which
words v ..v and u ..u we define the following value:
1 l 1 s
is used along with the “book stack” (or MTF) code (
[16], [17]) and some others (see for review ( [13], [15]). |ϕ(v1..vl/u1..us)|=|ϕ(u1..usv1..vl)|−|ϕ(u1..us)|,
In this paper we show how the idea of compression (1)
of combined texts described above can be used for where s,l are integers. Informally, it is the length of the
solving problems of homogeneity testing, classification, codewordforv1..vl ifitisencodedwiththewordu1..us.
and estimation of a measure of interdependence, or the
Theorem1. Letµ andµ bestationaryergodicsources
x y
association. A distinction of the suggested method from
generating letters from a finite alphabet A and let their
other approaches is that it belongs to the framework of
memory be upper-bounded by a certain constant M.
mathematical statistics.
Suppose that ...y y ... y y is generated by µ ,
−t −t+1 −1 0 y
II. DEFINITIONSANDPROBLEMFORMULATIONS whereas ...x−tx−t+1...x−1x0 and w1...wm,m≥1, are
generated by µ in such a way that all three sequences
First we briefly consider the main properties of so- x
are independent, and let ϕ be a universal code. Define
calleduniversalcodes,payingthemainattentiontotheir
meaning whereas formal definitions of codes, stationary ∆ =|ϕ(w ...w /y y ...y y )|−
t,k,m 1 m −k −k+1 −1 0
ergodic sources and Shannon entropy are given in Ap-
|ϕ(w ...w /x x ...x x )| (2)
pendix 1 and can be found in [5], [15]. 1 m −t −t+1 −1 0
Here we only note that we consider so-called lossless
where |ϕ(v ..v /u ..u )| is defined in (1). Then there
1 l 1 s
codes ϕ which encode words over alphabet A using
exists such a constant λ and an integer m that for any
0
wordsfromthebinaryalphabet{0,1}insuchawaythat
m > m there is such an integer L that, for any t >
0
any word can be decoded without mistakes, i.e. there
m+L, k >m+L,
exists a map ϕ−1 such that for any word w over A
ϕ−1(ϕ(w)) = w. Let a code ϕ be applied to encode EµxEµy(∆t,k,m) ≥ λm+◦(m), (3)
sequences generated by a stationary ergodic source µ.
and, if µ = µ then λ = 0, otherwise λ > 0. Here
The value R (ϕ,µ)= 1|ϕ(x ...x )| − h (µ) is called x y
t t 1 t ∞ E () is the expectation with respect to the measure ν.
the redundancy, where h (µ) is the limit entropy, see ν
∞
Appendix 1. (It is known that h (µ) is an attainable The proof is given in Appendix 2, whereas here
∞
lower limit for lossless codes, that is why the difference we give some explanations of the theorem. The
is called the redundancy.) By definition, the code ϕ is theorem compares two cases: the word w ...w is
1 m
called universal if the redundancy goes to 0 as t grows, compressed either together with x x ...x x
−t −t+1 −1 0
i.e. lim R (ϕ,µ) = 0. (In other words, a universal or with y y ...y y . The word w ...w is
t→∞ t −k −k+1 −1 0 1 m
code compresses sequences generated by any stationary generated by µ , hence, it should be compressed
x
ergodic source till the limit value.) better with x x ...x x and the value
−t −t+1 −1 0
The main goal of the paper is to give a compression- |ϕ(w ...w /x x ...x x )|(= |ϕ(x x
1 t −t −t+1 −1 0 −t −t+1
based solution for the following problems: ... x x w ...w )|− |ϕ(x x ...x x )|) should be
−1 0 1 t −t −t+1 −1 0
i)Homogeneitytest,wherethereareseveralsequences lessthan|ϕ(w ...w /y y ...y y )|.Thetheorem
1 t −k −k+1 −1 0
x1x1...x1 , ..., xkxk...xk , y1y1...y1 , ... ysys...ys , shows, that, indeed, the word w ...w is compressed
1 2 n1 1 2 nk 1 2 m1 1 2 ms 1 m
generated either by a single source or by two different better together with x x ...x x if the lengths
−t −t+1 −1 0
ones, and two corresponding hypotheses. We also con- of sequences w w ...w , x x ...x x and
1 2 m −t −t+1 −1 0
sider the more general case where there are more than y y ...y y are sufficiently large.
−k −k+1 −1 0
two different sets of sequences. Lettherebetwosetsofsequences:X ={x1x1...x1 ,
1 2 n1
ii) Classification problems, where there are samples x2x2...x2 , ..., xkxk...xk }, and Y = { y1y1...,y1 ,
1 2 n2 1 2 nk 1 2 m1
x1x1...x1 , ..., xkxk...xk , y1y1...y1 , ... ysys...ys y2y2...y2 , ... ysys...ys }, generated by possibly dif-
1 2 n1 1 2 nk 1 2 m1 1 2 ms 1 2 m2 1 2 ms
generated two different (but unknown) sources and ferentmeasuresµ andµ .Weconsidertwohypotheses
x y
z ...z is generated by one of the two. The goal is to H = {µ = µ } and H = {µ (cid:54)= µ } and our goal
1 l 0 x y 1 x y
determine which of them generated z ...z . is to develop a statistical test for them using the sets
1 l
iii) Estimation of a so-called measurement of interde- X and Y. First we give an informal description of the
pendence, or the association. suggestedtest,whichwillbebasedondatacompression.
2
CombineahalfofthesequencesfromthesetX intoX∗ y,y∗ ∈ Y the ratios |x|/|x∗| and |y|/|y∗| are upper-
(say, x1x1...x1 , x2x2...x2 , ... , x(cid:98)k/2(cid:99)x(cid:98)k/2(cid:99)...x(cid:98)k/2(cid:99) bounded by a certain constant, then, with probability 1,
1 2 n1 1 2 n1 1 2 n(cid:98)k/2(cid:99)
) and half of Y (say and y1y1...y1 , y1y2...y2 , ..., the Type II error goes to 0.
1 2 m1 1 2 m2
y(cid:98)s/2(cid:99)y(cid:98)s/2(cid:99)...y(cid:98)s/2(cid:99) ))intoY∗.Thencompressallother
1 2 m(cid:98)s/2(cid:99)
sequences using a universal code ϕ along with X∗ and The proof is given in Appendix 2, but here we give
Y∗. If H is true, the values |ϕ(xixi...xi / X∗)|, some comments. First, note that there are many tests
i = (cid:98)k/20(cid:99) + 1,...,k and |ϕ(yjyj...y1j 2/ Xni∗)|, j = of homogeneity for 2×2 tables and, in principle, any
(cid:98)s/2(cid:99)+1,...,sshouldbeevenlym1 i2xed.mOjtherwise,ifH of them can be used. That is why, we do not describe
1
is true, then, on average, the numbers from the second the test Ψα on sizes of n1,1,n1,2,n2,1,n2,2. Second,
set should be larger than those from the first one, be- the described method can be easily extended from the
causetheprobabilitydistributionsofthesequencesfrom two-sample problem to the s-sample problem, s > 2.
yiyi...yi andX∗ aredifferent,whereastheprobability Namely, let there be s,s > 2, sets V1, V2,..., Vs of
djistjributmiojnsofthesequencesxixi...xi andX∗ arethe sequencesgeneratedbystationaryergodicsourcesµ1,µ2
1 2 ni ...,µ . Let there be the hypotheses H = {µ = µ
same (here and below |U| is the number of elements in s 0 1 2
= ... = µ } and H = H¯ . In this case we carry
U). s 1 0
out calculations based on the scheme described above
The test. A more formal description of the suggested
test is as follows: i) Denote the set X \X∗ by Xˆ and in order to obtain a so-called s×s table and then apply
Y \Y∗ by Yˆ and calculate for any xixi...xi from Xˆ a test for homogeneity, see ( [7]).
1 2 ni
the values
Measurement of the interdependence and association.
γ =|ϕ(xixi...xi /Y∗)| − |ϕ(xixi...xi /X∗)| (4) If the hypothesis of homogeneity is rejected, it is
i 1 2 ni 1 2 ni natural to measure interdependence. We suggest to
and for any yjyj...yj from Yˆ measureinterdependencebetweentwosetsofsequences
1 2 mj
X and Y (and the corresponding sources) based on
δj =|ϕ(y1jy2j...ymj j/X∗)| − |ϕ(y1jy2j...ymj j/Y∗)|. (5) the above described 2 × 2 tables. (In the case of
the s-sample problem, s > 2, the measures will be
Define
based on the s × s table.) This problem is well-
n =|{i:γ ≥0}|, n =|{i:γ <0}|, investigated in the mathematical statistics, see, for
1,1 i 1,2 i
example, ( [7], part 33). That is why, we mention such
n =|{i:δ <0}|,n =|{i:δ ≥0}|. (6)
2,1 i 2,2 i measures only briefly. For 2×2 tables we mention the
coefficient of association, Q, defined by the equation
ii) Apply the test of the independence for the 2×2
Q = (n n − n n )/n n + n n )
table to 1,1 2,2 1,2 2,1 1,1 2,2 1,2 2,1
and the coefficient V = (n n − n n )/
1,1 2,2 1,2 2,1
N2,2 =(cid:12)(cid:12)(cid:12)(cid:12) nn12,,11 nn12,,22 (cid:12)(cid:12)(cid:12)(cid:12). (cid:112)see(,n(1,[17+], npa2r,2t)3(3n)1.,1It+isni2m,1p)o(nrt1a,n2t+tonn2o,2te)(tnh2a,t1t+henre2,2a)re,
well-known methods of building standard errors and
A detailed analysis of this problem is carried out in [7],
confidence interval for Q and V ( [7], part 33).
part 33. In particular, there is a description of efficient
testsforhomogeneityproblemforthe2×2table(seepart Let there be sequences wˆ1 = w1w1...w1 , wˆ2 =
1 2 m1
33.22). We denote this test as Ψ , where α is the level w2w2...w2 , ..., wˆk = wkwk...wk , generated by
α 1 2 m2 1 2 mk
of significance. Note, that there are some requirements stationary ergodic sources ν , ..., ν , correspondingly,
1 k
for values of n ,n ,n ,n which should be valid where k ≥ 2. There is a new sequence uˆ = u u ...u ,
1,1 1,2 2,1 2,2 1 2 n
if Ψ is used (see [7]). n ≥ 2, and it is known beforehand that it is generated
α
by one of the sources from {ν , ..., ν }. The problem
Theorem 2. Let µ and µ be stationary ergodic mea- 1 k
x y of classification is to determine which source generated
sureswhosememoryisfinite(but,possibly,unknown).If
the sequence uˆ=u u ...u . By definition, a method of
theabovedescribedtestisappliedfortestingH against 1 2 n
0 classification is called asymptotically consistent if, with
H along with the test Ψ and the requirements of Ψ
1 α α probability 1, the method finds ν which generated the
for values of n ,n ,n ,n are valid, then for any
1,1 1,2 2,1 2,2 sequence u u ...u when min(n,m ,m ,...,m ) goes
code ϕ the Type I error is not grater than α. 1 2 n 1 2 k
to infinity.
If ϕ is a universal code, |X| and |Y| go to infinity
and the lengths of all sequences from X and Y go to Method of classification. We suggest the follow-
infinity in such a way that for all sequences x,x∗ ∈X, ing method of classification: decide that the sequence
3
u u ...u was generated the source ν for which where P(b) and Q(b) are probability distributions over
1 2 n j
alphabet B. It is known that for any distributions P and
j =arg min |ϕ(u u ...u /wiwi...wi )|. (7)
i=1,...,k 1 2 n 1 2 mk Q the KL divergence is nonnegative and equals 0 if and
Theorem 3. Let ν , ..., ν be stationary ergodic mea- only if P(a)=Q(a) for all a.
1 k
sureswhosememoriesarefinite(but,possibly,unknown). Let us describe universal codes, or data compressors.
If ϕ is a universal code and the lengths of all sequences (A detailed description can be found, for example, in
uˆ, wˆ , wˆ , ..., wˆ go to infinity in such a way that [15].) First we note that (as is known in Information
1 2 k
Theory) sequences generated by a source p can be
lim |uˆ|/min|wˆj| = 0, (8)
”compressed” on average till the limit Shannon entropy
|uˆ|→∞ j
h (p) and, on the other hand, there is no code ψ for
then the described method is asymptotically consistent. ∞
whichtheexpectedcodewordlengthislessthanh (p);
∞
IV. APPENDIX1 see [5]. As defined above, a code ϕ is universal if it
Let τ be a stationary ergodic source generating letters compresses till this limit. We will consider universal
from a finite alphabet A. (Definitions can be found in codes which can be applied to any word x1...xs, n>0,
[5].) The m− order (conditional) Shannon entropy and from a certain alphabet A and such that the following
the limit Shannon entropy are defined as follows: natural property is valid: for any words u, v,|v|>0,
(cid:88) (cid:88)
h (τ)= − τ(v) τ(a|v)logτ(a|v), |ϕ(uv)|−|ϕ(u)|>0. (13)
m
v∈Am a∈A
For any universal code ϕ we define a measure π as
h (τ)= lim h (τ). (9) ϕ
∞ m→∞ m follows:
It is known that for any integer m
π (x x ...x ) = 2−|ϕ(x1x2...xt)|, (14)
ϕ 1 2 t
h (τ)≥h (τ) ≥h (τ), (10)
m m+1 ∞
where x x ...x is a word over A. It is known that for
1 2 t
see [5]. Now we define codes. Let A∞ be the set of any t
all infinite words x x ... over the alphabet A. A data (cid:88)
1 2 π (u) = 1, (15)
ϕ
compression method (or code) ϕ is defined as a set
u∈At
of mappings ϕ such that ϕ : An → {0,1}∗, n =
n n
1,2,... and for each pair of different words x,y ∈An (see the Kraft inequality and (11)).
ϕ (x) (cid:54)= ϕ (y). Informally, it means that the code ϕ Nowweconsideranapplicationofuniversalcodes(or
n n
can be applied for compression of each message of any datacompressors)whichisthemaintoolforsolvingthe
length n over the alphabet A and the message can be problems mentioned in Introduction. From (1) and (14)
decodedifitscodeisknown.Itisalsorequiredthateach we immediately obtain that
sequence ϕ (u )ϕ (u )...ϕ (u ),r ≥ 1, of encoded
n 1 n 2 n r |ϕ(v ..v /u ..u )|=−log(π (v ..v |u ..u )) (16)
words from the set An,n≥1, can be uniquely decoded 1 l 1 s ϕ 1 l 1 s
into u u ...u . Such codes are called uniquely decod- and the right part is defined correctly, see (13). The
1 2 r
able. For example, let A = {a,b}, the code ψ (a) = following property of universal codes will play an im-
1
0,ψ (b) = 00, obviously, is not uniquely decodable. It portant role in what follows. Let µ be a stationary
1 z
is well known that if a code ϕ is uniquely decodable ergodic source generating letters from an alphabet A
then the lengths of the codewords satisfy the following and ϕ be a universal code. For any integer m ≥ 1 and
inequality (Kraft inequality): Σ 2−|ϕ(u)| ≤ 1 , w ...w ∈Am with probability 1
u∈An 1 m
see, for ex., [5]. Moreover, if the sum Σ 2−|ϕ(u)|
u∈An π (w w ...w |z ...z )
is less than 1, there exists such a code ϕ∗ that i) lim log ϕ 1 2 m −T 0 =0. (17)
Σ 2−|ϕ∗(u)| = 1 , and |ϕ∗(u)| ≤ |ϕ(u)| for any T→∞ µz(w1w2...wm|z−T...z0)
u∈An
word u, [5]. (Informally, it means, that ϕ∗ compresses Note that this equation shows that the code ϕ estimates
better.) So, we can consider only codes for which the (unknown) probability precisely, where T grows.
Σ 2−|ϕ(u)| =1. (11)
u∈An V. APPENDIX2:PROOFS
We will use a so-called Kullback-Leibler (KL) diver- Proof of Theorem 1. First we prove the following
gence, which is defined by Claim.
(cid:88) P(b) (cid:88)
D(P||Q)= P(b)log , (12) E E µ (w w ...w )
Q(b) µx µy x 1 2 m
b∈B w1w2...wm
4
µ (w ...w |x x ...x ) (cid:88) (cid:88)
log x 1 m −t −t+1 0 =(m−M)λ+O(1), +...+ µx(w1...wM) µx(wM+1|w1...wM)
µ (w ...w |y y ...y )
y 1 m −k −k+1 0 (18) w1...wM wM+1
where M is a upper bound of memories of µx and µy, log µx((wM+1|w1...wM)
and λ is such a constant that λ = 0 if µx = µy and µy((wM+1|w1w2...wM)
λ>0 otherwise. (cid:88)
+E E ( µ (w w ...w )
Proof of the claim. The left side can be presented as µx µy x 1 2 M−1
follows: w1w2...wM−1
(cid:88)
(cid:88) µ (w |x w w ...w )
E E µ (w w ...w ) x M 0 1 2 M−1
µx µy x 1 2 m
w1w2...wm wM
µ ((w |x w w ...w )
log µx(w1...wm|x−tx−t+1...x0) = log µx((wM|y0w1w2...wM−1) +...
µ (w ...w |y y ...y ) y M 0 1 2 M−1
y 1 m −k −k+1 0
E E ( (cid:88) µ (w w ...w ) +(cid:88)µ (w )logµx(w1|x−M+1x−M+2...x0)). (20)
µx µy x 1 2 m−1 x 1 µ (w |y y ...y )
y 1 −M+1 −M+2 0
w1w2...wm−1 w1
(cid:88) For any function ψ(u ,...,u ) and any measure
µ (w |w w ...w ) e+1 e+b
x m 1 2 m−1
ν(u u ...u u ...u )
wm 1 2 e e+1 e+b
(cid:88)
log µx(wm|x−tx−t+1...x0w1w2...wm−1) ν(u1u2...ueue+1...ue+b)
µy(wm|y−ky−k+1...y0w1w2...wm−1) u1u2...ueue+1...ue+b
(cid:88) ψ(u ,...,u )
+ µ (w w ...w ) e+1 e+b
x 1 2 m−2
(cid:88)
w1w2...wm−2 = ν(ue+1...ue+b)ψ(ue+1,...,ue+b),
(cid:88) µ (w |w w ...w ) ue+1...ue+b
x m−1 1 2 m−2
where e and b are integers. Having taken into account
wm−1
this equation, stationarity µ and µ , equations (V), (V)
µ (w |x x ...x w w ...w ) x y
log x m−1 −t −t+1 0 1 2 m−2 and (V), we obtain
µ (w |y y ...y w w ...w )
y m−1 −k −k+1 0 1 2 m−2 (cid:88)
E E µ (w w ...w )
+...+(cid:88)µx(w1)logµµx((ww1||yx−txy−t+1......xy0))). (19) µx µyw1w2...wm x 1 2 m
y 1 −k −k+1 0
w1 log µx(w1...wm|x−tx−t+1...x0) =
It is supposed that the memory of µ and µ is µ (w ...w |y y ...y )
x y y 1 m −k −k+1 0
upper-bounded by M. Hence, by definition, it means (cid:88)
(m−M) µ (w w ...w )
that µ (v|u u ...u ) = µ (v|u u ...u ), x 1 2 M
r i i+1 j r j−M+1 j−M+2 j
wherej ≥i+M−1.Fromthisequationandtheprevious w1w2...wm−2
one we obtain the following equation: (cid:88)
µ (w |w w ...w )
x M+1 1 2 M
(cid:88)
EµxEµy µx(w1w2...wm) wM+1
w1w2...wm logµx((wM+1|w1w2...wM) +O(1).
log µx(w1...wm|x−tx−t+1...x0) = µy((wM+1|w1w2...wM)
µy(w1...wm|y−ky−k+1...y0) From properties of K-L divergence (see (12) ) we can
(cid:88) µ (w w ...w )(cid:88)µ (w |w w ...w ) see that λ > 0 if µx (cid:54)= µy and, obviously, λ = 0, if
x 1 2 m−1 x m 1 2 m−1 µ =µ . The claim is proven.
x y
w1w2...wm−1 wm Let us proceed with proof of the theorem. Having
µ (w |w ...w ) takenintoaccountthedefinitions(16)and(1),weobtain
log x m m−M m−1
µ (w |w ...w ) that
y m m−M m−1
E E (∆ )=
+ (cid:88) µ (w ...w ) (cid:88) µ (w |w ...w ) µx µy t,k,m
x 1 m−2 x m−1 1 m−2 (cid:88) (cid:88) (cid:88)
w1...wm−2 wm−1
µ (w ...w ) u−tu−t+1...u0∈At+1v−kv−k+1...v0∈Ak+1w1w2...um∈Am
log x m−M−1 m−2
µy(wm−M−1...wm−2) µy(v−kv−k+1...v0)µx(u−tu−t+1...u0)µx(w1w2...um)
5
π (w ...w |u u ...u )
log ϕ 1 m −t −t+1 0 . (21) [5] Cover, T. M., Thomas, J. A., 2006. Elements of information
πϕ(w1...wm|v−kv−k+1...v0) theory.Wiley-Interscience,NewYork,NY,USA.
[6] Ferragina,P.,Giancarlo,R.,Greco,V.,Manzini,G.,Valiente,G.,
The following equation is obvious: 2007. Compression-based classification of biological sequences
and structures via the universal similarity metric: experimental
π (w ...w |u u ...u )
log ϕ 1 m −t −t+1 0 = assessment.BMCbioinformatics8(1),1.
π (w ...w |v v ...v ) [7] Kendall,M.,Stuart,A.,1961.Theadvancedtheoryofstatistics;
ϕ 1 m −k −k+1 0
Vol.2:Inferenceandrelationship.London.
µ (w ...w |u u ...u ) [8] Khmelev,D.V.,Teahan,W.J.,2003.Arepetitionbasedmeasure
log x 1 m −t −t+1 0 + for verification of text collections and for text categorization.
µ (w ...w |v v ...v )
y 1 m −k −k+1 0 In: Proceedings of the 26th annual international ACM SIGIR
conferenceonResearchanddevelopmentininformaionretrieval.
π (w ...w |u u ...u )
log ϕ 1 m −t −t+1 0 + ACM,pp.104–110.
µ (w ...w |u u ...u ) [9] Kieffer, J. C., Yang, E.-H., 2000. Grammar-based codes: a new
x 1 m −t −t+1 0
class of universal lossless source codes. IEEE Transactions on
µ (w ...w |v v ...v ) InformationTheory46(3),737–754.
log y 1 m −k −k+1 0 . (22) [10] Kukushkina, O. V., Polikarpov, A., Khmelev, D. V., 2001. Us-
π (w ...w |v v ...v )
ϕ 1 m −k −k+1 0 ing literal and grammatical statistics for authorship attribution.
ProblemsofInformationTransmission37(2),172–184.
Thefirsttermisestimatedintheclaim,see(V),whereas
[11] Li,M.,Chen,X.,Li,X.,Ma,B.,Vita´nyi,P.M.,2004.Thesim-
thesecondandthethirdtermscanbeestimatedbasedon ilaritymetric.IEEEtransactionsonInformationTheory50(12),
(17). So, from the claim and (17), we can see that, with 3250–3264.
[12] Manzini,G.,2001.Ananalysisoftheburrowswheelertransform.
probability 1, E E (∆ ) = (m−M)λ+O(1).
µx µy t,k,m JournaloftheACM(JACM)48(3),407–430.
The theorem is proven. [13] Rissanen,J.,Jul.1984.Universalcoding,information,prediction,
Proof of the Theorem 2. First we consider the case andestimation.InformationTheory,IEEETransactionson30(4),
where H is true. It means that the sequences from Xˆ 629–636.
0 [14] Rissanen, J., Langdon, G. G., 1979. Arithmetic coding. IBM
and Yˆ obey the same distribution. Hence, γi (4) and Journalofresearchanddevelopment23(2),149–162.
δ (5) have the same distribution, too, and the above [15] Ryabko,B.,Astola,J.,Malyutov,M.,2016.Compression-Based
j
Methods of Statistical Analysis and Prediction of Time Series.
mentioned test Ψ from [7], part 33, can be applied.
α Springer.
Now we consider the case where H1 is true. In this [16] Ryabko,B.Y.,1980.Datacompressionbymeansofabookstack.
case the length of any sequence grows, so, the length ProblemyPeredachiInformatsii16(4),16–21.
[17] Ryabko,B.,Horspool,N.R.,Cormack,G.V.,Sekar,S.,Ahuja,
will be grater than m from Theorem 1. The number
0 S. B., 1987. Technical correspondence. Communications of the
of sequences grows to infinity and the total length of a ACM30(9),792–797.
half of them goes to infinity in such a way that for any [18] Teahan, W. J., Harper, D. J., 2003. Using compression-based
languagemodelsfortextcategorization.In:Languagemodeling
integer L the total length will be grater than the sum
forinformationretrieval.Springer,pp.141–165.
m+L from Theorem 1. From this theorem we can see [19] Vita´nyi, P. M., 2011. Information distance in multiples. IEEE
thatn andn goesto0and,hence,theTypeIIerror TransactionsonInformationTheory57(4),2451–2456.
1,2 2,1
[20] Ziv, J., Lempel, A., 1977. A universal algorithm for sequential
goes to 0.
data compression. IEEE Transactions on information theory
Proof of the Theorem 3. Suppose, that the sequence 23(3),337–343.
u u ...u was generated by ν . Then, we can see [21] Ziv,J.,Lempel,A.,1978.Compressionofindividualsequences
1 2 n j viavariable-ratecoding.IEEETransactionsonInformationThe-
from Theorem 1 that, with probability 1, the value
ory24,530–536.
|ϕ(u u ...u /wiwi...wi )| grows as λ n+◦(n), λ >
1 2 n 1 2 mk i i
0, if i (cid:54)= j, (i.e. wiwi...wi ) is generated by ν ). On
1 2 mi i
the other hand, |ϕ(u u ...u /wiwi...wi )| = ◦(n).
1 2 n 1 2 mk
Hence, |ϕ(u u ...u / wi wi ... wi )| is minimal when
1 2 n 1 2 mi
i = j (i.o., u u ...u is generated by ν ). The theorem
1 2 n j
is proven.
REFERENCES
[1] Bell, T., Witten, I. H., Cleary, J. G., 1989. Modeling for text
compression. ACM Computing Surveys (CSUR) 21 (4), 557–
591.
[2] Burrows,M.,Wheeler,D.J.,1994.Ablock-sortinglosslessdata
compressionalgorithm.
[3] Cilibrasi,R.,Vitanyi,P.,2005.Clusteringbycompression.IEEE
TransactionsonInformationTheory51,1523–1545.
[4] Cilibrasi, R., Vita´nyi, P., De Wolf, R., 2004. Algorithmic clus-
tering of music based on string compression. Computer Music
Journal28(4),49–67.
6