Algorithmic Information Theory: Mathematics of Digital Information

446 Pages·2006·3.71 MB·English

by  Seibt P.

SpringerSerieson
Signals and Communication Technology
Signals and Communication Technology
Algorithmic Information Theory RadioWave Propagation
Mathematics of Digital Information Processing for Telecommunication Applications
P. Seibt H. Sizun ISBN 3-540-40758-8
ISBN 3-540-33218-9
Electronic Noise and Interfering Signals
The Variational Bayes Method Principles and Applications
in Signal Processing G. Vasilescu ISBN 3-540-40741-3
V. Sˇm´ıdl and A. Quinn DVB
ISBN 3-540-28819-8 The Family of International Standards
Topics in Acoustic Echo and Noise Control for Digital Video Broadcasting, 2nd ed.
Selected Methods for the Cancellation of U. Reimers ISBN 3-540-43545-X
Acoustical Echoes, the Reduction of Digital Interactive TVand Metadata
Background Noise, and Speech Processing Future Broadcast Multimedia
E. Hänsler and G. Schmidt (Eds.) A. Lugmayr, S. Niiranen, and S. Kalli
ISBN 3-540-33212-x ISBN 3-387-20843-7
EM Modeling of Antennas and RF Components Adaptive Antenna Arrays
for Wireless Communication Systems Trends and Applications
F. Gustrau, D. Manteuffel S. Chandran (Ed.) ISBN 3-540-20199-8
ISBN 3-540-28614-4 Digital Signal Processing
Interactive Video with Field Programmable Gate Arrays
Algorithms and Technologies U. Meyer-Baese ISBN 3-540-21119-5
R. I Hammoud (Ed.) Neuro-Fuzzy and Fuzzy Neural Applications
ISBN 3-540-33214-6 in Telecommunications
Continuous-Time Signals P. Stavroulakis (Ed.) ISBN 3-540-40759-6
Y. Shmaliy SDMA for Multipath Wireless Channels
ISBN 1-4020-4817-3 Limiting Characteristics
and Stochastic Models
Voice and Speech Quality Perception
I.P. Kovalyov ISBN 3-540-40225-X
Assessment and Evaluation
U. Jekosch Digital Television
ISBN 3-540-24095-0 A Practical Guide for Engineers
W. Fischer ISBN 3-540-01155-2
Advanced Man-Machine Interaction
Fundamentals and Implementation Speech Enhancement
K.-F. Kraiss J. Benesty (Ed.)
ISBN 3-540-30618-8 ISBN 3-540-24039-X
Orthogonal Frequency Division Multiplexing Multimedia Communication Technology
for Wireless Communications Representation, Transmission
Y. Li (Ed.) and Identification of Multimedia Signals
ISBN 0-387-29095-8 J.R. Ohm ISBN 3-540-01249-4
Circuits and Systems Information Measures
Based on Delta Modulation Information and its Description in Science
Linear, Nonlinear and Mixed Mode Processing and Engineering
D.G. Zrilic ISBN 3-540-23751-8 C. Arndt ISBN 3-540-40855-X
Functional Structures in Networks Processing ofSAR Data
AMLn–ALanguageforModelDriven Fundamentals, Signal Processing,
Development of Telecom Systems Interferometry
T. Muth ISBN 3-540-22545-5 A. Hein ISBN 3-540-05043-4
continued after index
Peter Seibt
Algorithmic
Information Theory
Mathematics of Digital Information
Processing
14
With Figures
123
Peter Seibt
Université de la Méditerranée
and
Centre de Physique Théorique
Campus de Luminy, Case 907
13288 Marseille cedex 9, France
LibraryofCongressControlNumber:2006925851
ISSN 1860-4862
ISBN-103-540-33218-9SpringerBerlinHeidelbergNewYork
ISBN-13978-3-540-33218-3SpringerBerlinHeidelbergNewYork
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is
concerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broadcasting,
reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Duplicationofthispublicationor
partsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,1965,inits
currentversion,andpermissionforusemustalwaysbeobtainedfromSpringer-Verlag.Violationsareliable
toprosecutionundertheGermanCopyrightLaw.
SpringerisapartofSpringerScience+BusinessMedia.
springer.com
©Springer-VerlagBerlinHeidelberg2006
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnotimply,
evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevantprotectivelawsand
regulationsandthereforefreeforgeneraluse.
Typesetting by the authors and SPi
Coverdesign: Design & Production, Heidelberg
Printedonacid-freepaper SPIN: 11607311 62/2162/SPi 543210
Contents
1 Data Compaction.......................................... 5
1.1 Entropy Coding ......................................... 5
1.1.1 Discrete Sources and Their Entropy ................. 5
1.1.2 Towards Huffman Coding .......................... 10
1.1.3 Arithmetic Coding ................................ 32
1.2 Universal Codes: The Example LZW....................... 43
1.2.1 LZW Coding ..................................... 43
1.2.2 The LZW Decoder ................................ 45
2 Cryptography ............................................. 49
2.1 The Data Encryption Standard ........................... 50
2.1.1 The DES Scheme.................................. 50
2.1.2 The Cipher DES in Detail .......................... 53
2.2 The Advanced Encryption Standard: The Cipher Rijndael .... 60
2.2.1 Some Elementary Arithmetic ....................... 60
2.2.2 Specification of Rijndael............................ 77
2.2.3 The Key Schedule ................................. 86
2.2.4 Decryption with Rijndael........................... 92
2.3 The Public Key Paradigm and the Cryptosystem RSA ....... 93
2.3.1 Encryption and Decryption via Exponentiation........ 93
2.3.2 The Cryptosystem RSA ............................ 97
2.4 Digital Signatures .......................................101
2.4.1 Message Digests via SHA-1 .........................101
2.4.2 DSA: Digital Signature Algorithm ...................112
2.4.3 Auxiliary Algorithms for DSA ......................116
2.4.4 The Signature Algorithm rDSA .....................122
2.4.5 ECDSA – Elliptic Curve Digital Signatures ...........125
3 Information Theory and Signal Theory: Sampling and
Reconstruction ............................................171
3.1 The Discrete Fourier Transform ...........................172
VI Contents
3.1.1 Basic Properties...................................172
3.1.2 The Fast Fourier Transform Algorithm...............183
3.2 Trigonometric Interpolation...............................190
3.2.1 Trigonometric Polynomials .........................191
3.2.2 Sampling and Reconstruction .......................193
3.3 The Whittaker–Shannon Theorem .........................198
3.3.1 Fourier Series .....................................198
3.3.2 The Whittaker–Shannon Theorem for Elementary
Periodic Functions.................................203
3.3.3 The (Continuous) Fourier Transform: A Sketch........209
3.3.4 The Sampling Theorem ............................214
4 Error Control Codes.......................................221
4.1 The Reed–Solomon Codes ................................221
4.1.1 Preliminaries: Polynomial Codes ....................221
4.1.2 Reed–Solomon Codes ..............................225
4.2 Convolutional Codes.....................................239
4.2.1 Encoding: Digital Filtering in Binary Arithmetic ......239
4.2.2 Decoding: The Viterbi Method......................253
5 Data Reduction: Lossy Compression .......................267
5.1 DFT, Passband Filtering and Digital Filtering ..............268
5.2 The Discrete Cosine Transform............................274
5.2.1 Functional Description of the DCT ..................275
5.2.2 The 2D DCT .....................................293
5.2.3 The Karhunen–Lo`eve Transform and the DCT ........305
5.3 Filter Banks and Discrete Wavelet Transform ...............314
5.3.1 Two Channel Filter Banks..........................314
5.3.2 The Discrete Wavelet Transform ....................372
References.....................................................435
Index..........................................................439
Introduction
Shall we be destined to the days of eternity, on holy-days,as well as
working days, to be shewing the RELICKS OF LEARNING, as monks do
the relicks of their saints – without working one – one single miracle
with them?
Laurence Sterne, Tristram Shandy
This book deals with information processing; so it is far from being a book
on information theory (which would be built on description and estimation).
The reader will be shown the horse, but not the saddle. At any rate, at the
verybeginning,therewasaseriesoflectureson“Informationtheory,through
the looking-glass of an algebraist”, and, as years went on, a steady process of
teaching and learning made the material evolve into the present form. There
still remains an algebraic main theme: algorithms intertwining polynomial
algebra and matrix algebra, in the shelter of signal theory.
AsolidknowledgeofelementaryarithmeticandLinearAlgebrawillbethe
key to a thorough understanding of all the algorithms working in the various
bit-stream landscapes we shall encounter. This priority of algebra will be the
thesis that we shall defend. More concretely: We shall treat, in five chapters
of increasing difficulty, five sensibly different subjects in Discrete Mathemat-
ics.Thefirsttwochaptersondatacompaction(losslessdatacompression)and
cryptographyareonanundergraduatelevel–themostdifficultmathematical
prerequisite will be a sound understanding of quotient rings, especially of fi-
nitefields(mostlyincharacteristic2).Thenexttwochaptersarealreadyona
graduatelevel;thereadershouldbeslightlyacquaintedwithargumentsinsig-
nal theory – although Lebesque integration could remain the “grey box” that
it usually is. We encounter sampling – an innocent operation of tremendous
epistemologicalimpact:thePlatonicmathematicianleavinghisheavenofcon-
tinuity(rule=truth)fortheearthofdiscreteness(diversity=uncertainty)will
beplainlycomfortedbythegreatinterpolationtheoremsthatlifthimbackto
the heights. The chapter on error control codes which are designed according
to signal theoretical ideas, complements – on a purely algebraic level – the
invasion of signal theory. The fifth and final chapter is the most important,
in length as well as in complexity. It deals with lossy (image) compression,
and yields the mathematical background for the understanding of JPEG and
JPEG2000.Now,ourPlatonicmathematicianwillbeexpelledfromparadise:
The discrete world becomes absolute, and all continuous constructions are
plainly auxiliary and relative.
But let us pass to a detailed description of the content.
The first chapter on data compaction is more or less an elementary intro-
ductiontoalgorithmicinformationtheory.Thecentralthemewillbethenon-
redundant representation of information. Everything turns around the notion
ofentropy:Whatistheinformationcontentofastringofsymbols(withgiven
2 Introduction
statisticalbehaviour),i.e.whatisitsminimalbitequivalent?Entropycoding
has its algorithmic stars: for memoryless sources, Huffman entropy coding is
unbeatable, but from a dynamic viewpoint, arithmetic coding will be slightly
better. Both methods are plainly integrated in advanced image compression
standards – we shall give a “default” Huffman table for JPEG. The chapter
will end with an (merely descriptive) exposition of the algorithm LZW which
isuniversalinthesensethatitcompactsanycharacterstream–withoutpre-
liminary statistical evaluation – by establishing a dictionary that enumerates
typical substrings (thereby creating its proper statistical evaluation). LZW is
theperfectdatacompactionalgorithm–butitneedslargefilesinordertobe
efficient. That is why we do not meet it in image compression where the data
units are too small.
The second chapter presents a set of rather austere lectures on cryptogra-
phy. We aim to give the maximum of information in a minimum of space –
there already exists a lot of highly coloured frescoes on the subject in print.
The venerable algorithm DES – the cryptosystem the best understood on
planet earth – will serve as an introduction to the subject. Things become
more serious with the new standard AES-Rijndael, the mathematical base-
mentofwhichisanicechallengetothestudent’sunderstandingofhigherlevel
(still)elementaryarithmetic.Hewilllearntothinkincyclicarithmetic – thus
getting familiar with discrete logarithms in a very explicit way. This opens
thedoortodigitalsignatures,i.e.tothepracticalrealizationofthepublickey
paradigm: I tell you my position on an arithmetic circle, but I do not reveal
the number of steps to get there. We shall treat the principal standard for
digital signatures, the system DSA (Digital Signature Algorithm), as well as
the variants rDSA (signatures via RSA) and ECDSA (signatures via elliptic
curvearithmetic).AstoRSA:Thisthirty-year-oldalgorithmhasalwaysbeen
the cornerstone of academic zest to promote the public key idea. So we shall
follow tradition – not without noting that RSA is a little bit old fashioned.
Finally, the secure hash algorithm (SHA-1) will produce the message digests
used in the various signature protocols. We shall need a lot of large prime
numbers; hence we include a brief discussion on their efficient generation.
Thiscompletesthedescriptionoftheeasypartofthisbook.Teachingexpe-
rienceshowsthatstudentslikedatacompactionforitssimpleeleganceandare
distant towards the iterative flatness of most cryptographic standards – are
they to blame?
With the third chapter, we enter the mathematical world of signal theory.
We have to answer the question: What is the discrete skeleton of a (con-
tinuous) signal? This means sampling, and reconstruction via interpolation.
Putting aside all practical considerations, we shall treat the problem in vitro.
Tough mathematical expositions are available; we have chosen a step-by-step
approach.So,webeginwiththediscreteFouriertransformanditsimportance
for trigonometric interpolation. Then we show ad hoc the classical interpola-
tion theorem (of Whittaker–Shannon, Nyquist–Shannon, or simply Shannon,
as you like it...) for precisely trigonometric polynomials. Finally, we attack
Introduction 3
the interpolation theorem in its usual form. There are some formal prob-
lems which need a short commentary. The natural mathematical framework
for signal theory is the L2 Hilbert space formalism. Now, elements of an L2
space are not functions (which disappear in their clouds of equivalence) but
function behaviour sketches. Precise numerical rules enter via duality. Thus,
sampling – which is basically a Hilbert space nonsense – must be considered
as a rule of behaviour (and should be duly formalized by a distribution). The
equality in the Shannon interpolation formula (which means equality of dis-
tributions) is, in any down-to-earth exposition, considerably fragilized by the
proof that establishes it. We shall try to be as simple as possible, and avoid
easy “distribution tricks”.
Logically, it is the fifth and last chapter on data compression that should
nowfollow.Whythisstrangedetourinthelandoferrorcontrolcodes?There
are at least two reasons. First, we get an equilibrium of complementary lec-
tures, when alternating between non-algebraic and algebraic themes. Then,
thefourthchapterlogicallyreinforcesourdefinitesubmissiontosignaltheory.
The codes of Reed–Solomon – our first subject – have a nice error- correcting
algorithm that makes use of the Discrete Fourier Transform over finite fields
of characteristic 2. And the convolutional codes – our second subject – are
bestunderstoodviadigitalfilteringinbinaryarithmetic.Ourexpositionthere
is non-standard, with a neat accent on algorithmic questions (no trellis nor
finite automata formalisms).
Finally,wecometothefifthchapter,whichisrathervoluminousandtreats
data compression, i.e. the practice of intentionally reducing the information
content of a data record – and this in such a way that the reproduction has
as little distortion as possible. We shall concentrate on image compression, in
particular on JPEG and JPEG 2000. The quality of compression depends on
sifting out efficiently what is considered to be significant numerical informa-
tion. Quantization towards bit representation will then annihilate everything
that can be neglected.
Our main concern will be to find an intelligent information theoretic sieve
method.
It is the Discrete Cosine Transform (DCT) in JPEG, and the Discrete
Wavelet Transform (DWT) in JPEG 2000 that will resolve our problems.
In both cases, a linear transformation will associate with regions of digital
imagesamples(consideredasmatricesofpicturalmeaning)matrixtransforms
whose coefficients have no longer a pictorial but only a descriptive meaning.
Wemustinsist:Ourtransformations willnotcompressanything;theymerely
will arrange the numerical data in a transparent way, thus making it possible
to define sound quantization criteria for efficient suppression of secondary
numerical information.
We shall begin the fifth chapter with a slight non-thematic digression:
the design of digital passband filters in a purely periodic context. This will
be a sort of exercise for formally correct thinking in the sequel. Then we
come up with the discrete cosine transform and its raison d’ˆetre in JPEG.