Linear Algebra, Signal Processing, and Wavelets - A Unified Approach: Python Version

386 Pages·2019·14.27 MB·English

by  Øyvind Ryan

Springer Undergraduate Texts
in Mathematics and Technology
Øyvind Ryan
Linear Algebra,
Signal Processing,
and Wavelets -
A Unifi ed Approach
Python Version
Springer Undergraduate Texts
in Mathematics and Technology
SeriesEditors:
HelgeHolden,NorwegianUniversityofScienceandTechnology,Trondheim,Norway
KeriA.Kornelson,UniversityofOklahoma,Norman,OK,USA
EditorialBoard:
LisaGoldberg,UniversityofCalifornia,Berkeley,CA,USA
ArminIske,UniversityofHamburg,Hamburg,Germany
PalleE.T.Jorgensen,TheUniversityofIowa,IowaCity,IA,USA
Springer UndergraduateTextsin MathematicsandTechnology(SUMAT)publishestext-
books aimed primarily at the undergraduate. Each text is designed principally for students
who are considering careers either in the mathematical sciences or in technology-based
areas such as engineering,finance, informationtechnologyand computerscience, bioscience
andmedicine,optimizationorindustry.Textsaimtobeaccessibleintroductionstoawiderange
ofcoremathematicaldisciplinesandtheirpractical,real-worldapplications;andarefashioned
bothforcourseuseandforindependentstudy.
Moreinformationaboutthisseriesathttp://www.springer.com/series/7438
Øyvind Ryan
Linear Algebra, Signal
Processing, and Wavelets -
A Unified Approach
Python Version
123
ØyvindRyan
DepartmentofMathematics
UniversityofOslo
Oslo,Norway
TheauthorhasreceivedfundingfromtheNorwegianNon-FictionWritersandTranslatorsAssociation.
Additionalmaterialtothisbookcanbedownloadedfromhttp://extras.springer.com.
ISSN1867-5506 ISSN1867-5514 (electronic)
SpringerUndergraduateTextsinMathematicsandTechnology
ISBN978-3-030-02939-5 ISBN978-3-030-02940-1 (eBook)
https://doi.org/10.1007/978-3-030-02940-1
LibraryofCongressControlNumber:2018964080
©SpringerNatureSwitzerlandAG2019
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broadcasting,reproduction
onmicrofilmsorinanyotherphysicalway,andtransmissionorinformationstorageandretrieval,electronicadaptation,
computersoftware,orbysimilarordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublicationdoesnot
imply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevantprotectivelawsand
regulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbookarebelieved
tobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsortheeditorsgiveawarranty,
expressorimplied,withrespecttothematerialcontainedhereinorforanyerrorsoromissionsthatmayhavebeenmade.
Thepublisherremainsneutralwithregardtojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
To Natalia, Katrina, and Alina.
Preface
TheideaforthisbookbeganwithanewcourseattheUniversityofOslo,called“Appli-
cationsofLinearAlgebra”,whichwasofferedforthefirsttimein2012.Attheuniversity
we had come to realize that students who have had a first course in linear algebra, al-
ready have the prerequisites to learn about severalimportant and interesting topics in
signal processing and wavelet theory. Unfortunately, most textbooks on these subjects
are written in a language poorly suited to a linear algebra background, making much
literature only accessible to engineering and signal processing students, and excluding
a large number of students. It also seems that it is not a common textbook strategy
to introduce signalprocessingandwaveletstogether fromscratch,eventhoughthe two
can very much motivate one another. Why not write a self-contained textbook, where
linear algebra is the main foundation? This notion is what motivated the preparation
of this book.
The author must admit that he is not particularly fond of the majority of signal
processingliterature,whichoftenhides ideasandresults whichessentiallyboildownto
basic linear algebra, using a signal processing nomenclature that is difficult for many
mathematicians to understand. Some examples are:
• Matrix notation is often absent. Instead, linear operations, such as the DFT, are
often expressed by component formulas. Matrix multiplication is absent, although
basicoperationssuchasconvolutioncanbeinterpretedasthis.Asaresult,important
matrix factorizations are not included.
• Many operationsthat representchangesof coordinates(such as the DFT, DCT and
DWT) are not represented as such. As a result, no tools, notation or results from
linear algebra are used for these operations.
• Eigenvalues and eigenvectors are not mentioned, even when these concepts could
shed light on the theory: It is often not mentioned that the Fourier basis vectors
are eigenvectors for filters, with the frequency response being the corresponding
eigenvalues. The property that convolution in time corresponds to multiplication in
frequency, an important notion in signal processing, can be summarized as follows
in linear algebra: for matrices with the same eigenvectors, the eigenvalues of the
product are the product of the eigenvalues.
• Function spaces are rarely put into the context of vector or inner product spaces,
evenif Fourier seriescanbe seen asa leastsquaresapproximationfromsuch spaces.
VII
VIII Preface
Further, it is rarely mentioned that the formulas for the Fourier coefficients (the
Fourier integrals) follow from the orthogonaldecomposition theorem.
Others have also recognized the need to write new textbooks that employ more linear
algebra. As an example, book [70] goes further in using matrix notation than many
signal processing textbooks. Still, more can be done in this direction, so that students
with only a basic linear algebra backgroundwill feel more at home.
This book provides anintroduction to Fourier analysis and signalprocessing (in the
first part of the book) and wavelets (in the second), assuming readers have completed
anintroductorycourseinlinearalgebra.Withoutthisbackgroundknowledge,thebook
willbe oflimitedvalue.Anappendixhasbeenincludedsothatstudentscanreviewthe
most important linear algebra concepts, but a full course on these topics is preferred
in order to follow the book. We use book [32] as the primary reference work for linear
algebra,meaningthatmanyconcretedefinitionsandconceptsfromthatbookarecited
by concreteidentifiers(i.e. pagenumber,theoremnumber,etc.). This isa verypopular
choice for a first course in linear algebra,but there are many other excellent textbooks
available. Leon [33] is a good example, which, just as [32], includes many exercises
related to MATLAB. Several other good introductions to linear algebra claim to be
applied in some way,such as [53], [75] and [69]. For a book that goes a bit further into
the theory than these, please consult [49] or [37].
The book can be used at the undergraduate level, and can be taken directly after
a first course in linear algebra. It is also possible to use the material at the graduate
level. A number of theory aspects from basic linear algebra are further developed:
• Complex vectorspaces andinner products areconsidered(many introductorylinear
algebra textbooks concentrate only on realvector spaces and inner product spaces).
• Inner product spaces that are function spaces are used extensively. Many introduc-
tory linear algebra textbooks consider such spaces, but often fail to offer extensive
practice with them.
• More intuition on changes of coordinates is developed, in particular for the DFT,
DCT and DWT.
The book itself can be viewed as an extension to a basic linear algebra textbook, and
may in the future become additional chapters in such a book.
Thestyleofthebookisverydifferentfrommosttextbooksonsignalprocessing,since
thelanguageoflinearalgebraisusedconsistently.Muchofthematerialandmanyproofs
havebeenredesignedfromtheirsourcestofitintoalinearalgebracontext.Thematerial
on wavelets has also been redesigned for this purpose, but also because it is not that
common to introduce wavelets at the undergraduate level. The book also attempts to
breakupmuchtheorywithexplanatorycode.Thisapproach,too,israrelyfoundinthe
literature,where code is often movedto appendices, to separateit completely fromthe
theory.Theauthorbelievesthatsuchseparationisunfortunateinmanycases,asitcan
lead students to spend all their time on the theory alone. Due to these considerations,
this book has been more than 6 years in the making.
Sincethebookdoesnotofferacomprehensivetreatmentofallbasicsignalprocessing
concepts or nomenclature, some engineering students may feel excluded. To make the
bookmoreaccessibleforsuchstudents,asectionisincludedtohelpreconcilethe signal
processing perspective and linear algebra perspective. The summaries throughout the
book also address the connections and differences between the two perspectives. The
section below on how to use the book in engineering courses provides further details.
Preface IX
The book attempts to go further than many books with the name “applications”
in their title, in that it has a clear computational perspective. The theory motivates
algorithmsandcode,forwhichmanyprogrammingandbestcodingpracticeissuesneed
tobeaddressed.Thebookfocusesonintegratingapplications,suchasmodernstandards
for the compression of sound and images (MPEG, JPEG, JPEG2000). A public and
open source github repository for Fourier analysis and wavelets accompanies the book,
whereallcodefromthebookcanbefound(inparticulartheFFTandDWTlibraries),as
wellasaccompanyingtestcode,documentation,andsamplefiles.Throughoutthebook
this repositorywillsimply be referredto as“the library”.The addressofthe repository
is https://github.com/oyvindry/applinalgcode. The library contains more code than
showninthebook,andthiscodewillbecontinuouslyupdatedandimproved.Notealso
that the scope of the library is larger than that of the book. As an example, a paper
on more general boundary modes for wavelets is being completed in parallel with the
book. Also, since most algorithms in the book are well suited for parallel computing,
future versions of the library and the book may support GPU-based programming.
The library differs from many toolboxes in that the individual steps in a given im-
plementation are firmly anchored in labeled formulas in the book. The focus on the
computationalperspectivehasbeeninspiredbythe project“Computing inScienceEd-
ucation”attheUniversityofOslo,aninitiativetointegratecomputationsintothebasic
science curriculum at the university from the very first semester.
Programming
Itisassumedthatthestudenthasalreadybeenintroducedtoaprogramminglanguage
or computational tool. Preferably, he or she will have already taken a full course in
programming first, since the book does not include an introduction to primitives such
as loops, conditional statements, lists, function definitions, file handling, or plotting.
At the University of Oslo, most students take such a Python-based course during the
first semester, focusing on such primitives. This course uses book [31], which provides
an excellent introduction to Python programming for beginning students.
Thisbookcomesintwoversions:onebasedonMATLAB,andoneonPython(where
version3isthesupportedversion).TheversionofthebookyouarereadingusesPython.
Thereareseveralotherrecentexamplesofbooksadaptedtodifferentlanguages:See[41]
and[42]forMATLABandPythonversionsofatextonmechanics,and[34]and[35]for
MATLAB andPythonversionsofacomputationalsciencetextbook(thelatterarealso
recommendedintroductionstoprogrammingintheselanguagesforbeginningstudents).
If you search the internet for recommendations about what programming language to
use in a basic linear algebra course, you may find statements such as “Python is too
advanced for such a beginning course”, or “You can get started much quicker with
MATLAB”. A good reply to such statements may be that the choice of programming
language should most likely depend on how programming is already integrated into
the courses at your university. Provided the integration at your university is thought-
through and well-organized, the programming in the book should not prove to be too
advanced for you, regardless of the choice of language.
Todistinguishbetweenthelibraryandothertoolboxes,mostfunctionsinthelibrary
havetheending"_impl".IfyoucomparethecodeintheMATLABandPythonversions
of the book, you will see that the code in the two languages is very similar:
X Preface
• Functionsignaturesandvariablenamesarevirtuallythesame,followingthePython
standard (words separated by "_" and using lowercase letters).
• Code indentation follows the Python standard, where it is an important part of the
syntax.
• The heavily used Python packagenumpydoes not use a prefix, since MATLAB does
not prefix code (this means that other packages with conflicting names must use a
prefix). For the same reason, the modules developed in the book also do not use a
prefix.
There are also some differences in the MATLAB and Python versions, however.
• The Python code is split into modules, a very important structuring concept in
Python. The book explains which modules the corresponding code is part of. In
MATLAB there is no module concept,andmany functions mustbe placedinfiles of
the samename. As a result,there aremore files in the MATLAB partofthe library.
• In Python, it is customary to place test code in the same module as the functions
being tested. Since MATLAB does not use modules, separate files are instead used
for test code instead.
• MATLAB passes all input and return parameters by value, not by reference, as
languages like Python do. This means that we can perform in-place computation in
Python,i.e.theresultcanbewrittendirectlyintotheinputbuffer,avoidingmemory
copy. This can lead to much more efficient code, so the Python code performs in-
place operations wherever possible. This affects the signatures of many functions:
Several Python functions have no return values, since the result is written directly
into the input buffer, while the MATLAB counterparts use a return value for the
result. Examples of in-place algorithms developed in this book are the DFT, the
DCT, filters, the DWT, and filter bank transforms.
There are also many other small differences between the two languages:
• Indicesinvectorsstartwith1inMATLAB,and0inPython(indexingstartingwith
0 is more natural for many of the algorithms discussed in the book).
• Thedistinctionbetweendifferentdimensions(1,2,3,andsoon)isclearerinPython.
• Extracting subsets of an object corresponding to rows/columns with a given set of
indices works more uniformly in MATLAB.
Many of these differences can be easily overcome, however, and there are many trans-
lation guides available online.1
MATLAB started as a command-based tool, and has evolved over the years to sup-
port many features of modern programming languages. Python on the other hand was
designed as a generalprogramminglanguage fromthe outset. As a result,MATLAB is
not as structured as Python, and there are severalbad MATLAB programming habits
around, which produce programs with illogical structures. We have sought to follow
many Python conventions for MATLAB code, in an attempt to avoid many of these
habits.
Python has support for classes, but their use in the library has been limited for
two reasons. First of all, class hierarchies don’t greatly simplify programming of the
concepts in the book. Secondly, although MATLAB has some primitive support for
classes, it remains unclear to the author how well this support actually works. By not
using classes, the MATLAB and Python code in the library have virtually identical
APIs.
1 Seeforinstancehttp://mathesaurus.sourceforge.net/matlab-numpy.html.