Complex Valued Nonlinear Adaptive Filters. Noncircularity, Widely Linear and Neural Models

322 Pages·2009·8.192 MB·English

by  Danilo P. Mandic, Vanessa Su Lee Goh

Complex Valued
Nonlinear Adaptive Filters
Noncircularity, Widely Linear
and Neural Models
DaniloP.Mandic
ImperialCollegeLondon,UK
VanessaSuLeeGoh
ShellEP,Europe
Thiseditionfirstpublished2009
© 2009,JohnWiley&Sons,Ltd
LibraryofCongressCataloging-in-PublicationData
Mandic,DaniloP.
Complexvaluednonlinearadaptivefilters:noncircularity,widelylinear,andneuralmodels/byDaniloP.
Mandic,VanessaSuLeeGoh,ShellEP,Europe.
p.cm.
Includesbibliographicalreferencesandindex.
ISBN978-0-470-06635-5(cloth)
1. Functionsofcomplexvariables.2. Adaptivefilters–Mathematicalmodels.3. Filters(Mathematics)
4. Nonlineartheories.5. Neuralnetworks(Computerscience)I. Goh,VanessaSuLee.II. Holland,Shell.
III. Title.
TA347.C64.M362009
621.382’2–dc22 2009001965
ISBN:978-0-470-06635-5
Typesetin10/12ptTimesbyThomsonDigital,Noida,India
PrintedinGreatBritainbyCPIAntonyRowe,Chippenham,Wiltshire
Contents
Preface xiii
1 TheMagicofComplexNumbers 1
1.1 HistoryofComplexNumbers 2
1.1.1 HypercomplexNumbers 7
1.2 HistoryofMathematicalNotation 8
1.3 DevelopmentofComplexValuedAdaptiveSignalProcessing 9
2 WhySignalProcessingintheComplexDomain? 13
2.1 SomeExamplesofComplexValuedSignalProcessing 13
2.1.1 DualityBetweenSignalRepresentationsinRandC 18
2.2 ModellinginCisNotOnlyConvenientButAlsoNatural 19
2.3 WhyComplexModellingofRealValuedProcesses? 20
2.3.1 PhaseInformationinImaging 20
2.3.2 ModellingofDirectionalProcesses 22
2.4 ExploitingthePhaseInformation 23
2.4.1 SynchronisationofRealValuedProcesses 24
2.4.2 AdaptiveFilteringbyIncorporatingPhaseInformation 25
2.5 OtherApplicationsofComplexDomainProcessingofRealValuedSignals 26
2.6 AdditionalBenefitsofComplexDomainProcessing 29
3 AdaptiveFilteringArchitectures 33
3.1 LinearandNonlinearStochasticModels 34
3.2 LinearandNonlinearAdaptiveFilteringArchitectures 35
3.2.1 FeedforwardNeuralNetworks 36
3.2.2 RecurrentNeuralNetworks 37
3.2.3 NeuralNetworksandPolynomialFilters 38
3.3 StateSpaceRepresentationandCanonicalForms 39
4 ComplexNonlinearActivationFunctions 43
4.1 PropertiesofComplexFunctions 43
4.1.1 SingularitiesofComplexFunctions 45
4.2 UniversalFunctionApproximation 46
4.2.1 UniversalApproximationinR 47
4.3 NonlinearActivationFunctionsforComplexNeuralNetworks 48
4.3.1 Split-complexApproach 49
4.3.2 FullyComplexNonlinearActivationFunctions 51
4.4 GeneralisedSplittingActivationFunctions(GSAF) 53
4.4.1 TheCliffordNeuron 53
4.5 Summary:ChoiceoftheComplexActivationFunction 54
5 ElementsofCRCalculus 55
5.1 ContinuousComplexFunctions 56
5.2 TheCauchy–RiemannEquations 56
5.3 GeneralisedDerivativesofFunctionsofComplexVariable 57
5.3.1 CRCalculus 59
5.3.2 LinkbetweenR-andC-derivatives 60
5.4 CR-derivativesofCostFunctions 62
5.4.1 TheComplexGradient 62
5.4.2 TheComplexHessian 64
5.4.3 TheComplexJacobianandComplexDifferential 64
5.4.4 GradientofaCostFunction 65
6 ComplexValuedAdaptiveFilters 69
6.1 AdaptiveFilteringConfigurations 70
6.2 TheComplexLeastMeanSquareAlgorithm 73
6.2.1 ConvergenceoftheCLMSAlgorithm 75
6.3 NonlinearFeedforwardComplexAdaptiveFilters 80
6.3.1 FullyComplexNonlinearAdaptiveFilters 80
6.3.2 DerivationofCNGDusingCRcalculus 82
6.3.3 Split-complexApproach 83
6.3.4 DualUnivariateAdaptiveFilteringApproach(DUAF) 84
6.4 NormalisationofLearningAlgorithms 85
6.5 PerformanceofFeedforwardNonlinearAdaptiveFilters 87
6.6 Summary:ChoiceofaNonlinearAdaptiveFilter 89
7 AdaptiveFilterswithFeedback 91
7.1 TrainingofIIRAdaptiveFilters 92
7.1.1 CoefficientUpdateforLinearAdaptiveIIRFilters 93
7.1.2 TrainingofIIRfilterswithReducedComputational
Complexity 96
7.2 NonlinearAdaptiveIIRFilters:RecurrentPerceptron 97
7.3 TrainingofRecurrentNeuralNetworks 99
7.3.1 OtherLearningAlgorithmsandComputationalComplexity 102
7.4 SimulationExamples 102
8 FilterswithanAdaptiveStepsize 107
8.1 BenvenisteTypeVariableStepsizeAlgorithms 108
8.2 ComplexValuedGNGDAlgorithms 110
8.2.1 ComplexGNGDforNonlinearFilters(CFANNGD) 112
8.3 SimulationExamples 113
9 FilterswithanAdaptiveAmplitudeofNonlinearity 119
9.1 DynamicalRangeReduction 119
9.2 FIRAdaptiveFilterswithanAdaptiveNonlinearity 121
9.3 RecurrentNeuralNetworkswithTrainableAmplitudeofActivation
Functions 122
9.4 SimulationResults 124
10 Data-reusingAlgorithmsforComplexValuedAdaptiveFilters 129
10.1 TheData-reusingComplexValuedLeastMeanSquare(DRCLMS)
Algorithm 129
10.2 Data-reusingComplexNonlinearAdaptiveFilters 131
10.2.1 ConvergenceAnalysis 132
10.3 Data-reusingAlgorithmsforComplexRNNs 134
11 ComplexMappingsandMo¨biusTransformations 137
11.1 MatrixRepresentationofaComplexNumber 137
11.2 TheMo¨biusTransformation 140
11.3 ActivationFunctionsandMo¨biusTransformations 142
11.4 All-passSystemsasMo¨biusTransformations 146
11.5 FractionalDelayFilters 147
12 AugmentedComplexStatistics 151
12.1 ComplexRandomVariables(CRV) 152
12.1.1 ComplexCircularity 153
12.1.2 TheMultivariateComplexNormalDistribution 154
12.1.3 MomentsofComplexRandomVariables(CRV) 157
12.2 ComplexCircularRandomVariables 158
12.3 ComplexSignals 159
12.3.1 WideSenseStationarity,Multicorrelations,andMultispectra 160
12.3.2 StrictCircularityandHigher-orderStatistics 161
12.4 Second-orderCharacterisationofComplexSignals 161
12.4.1 AugmentedStatisticsofComplexSignals 161
12.4.2 Second-orderComplexCircularity 164
13 WidelyLinearEstimationandAugmentedCLMS(ACLMS) 169
13.1 MinimumMeanSquareError(MMSE)EstimationinC 169
13.1.1 WidelyLinearModellinginC 171
13.2 ComplexWhiteNoise 172
13.3 AutoregressiveModellinginC 173
13.3.1 WidelyLinearAutoregressiveModellinginC 174
13.3.2 QuantifyingBenefitsofWidelyLinearEstimation 174
13.4 TheAugmentedComplexLMS(ACLMS)Algorithm 175
13.5 AdaptivePredictionBasedonACLMS 178
13.5.1 WindForecastingUsingAugmentedStatistics 180
14 DualityBetweenComplexValuedandRealValuedFilters 183
14.1 ADualChannelRealValuedAdaptiveFilter 184
14.2 DualityBetweenRealandComplexValuedFilters 186
14.2.1 OperationofStandardComplexAdaptiveFilters 186
14.2.2 OperationofWidelyLinearComplexFilters 187
14.3 Simulations 188
15 WidelyLinearFilterswithFeedback 191
15.1 TheWidelyLinearARMA(WL-ARMA)Model 192
15.2 WidelyLinearAdaptiveFilterswithFeedback 192
15.2.1 WidelyLinearAdaptiveIIRFilters 195
15.2.2 AugmentedRecurrentPerceptronLearningRule 196
15.3 TheAugmentedComplexValuedRTRL(ACRTRL)Algorithm 197
15.4 TheAugmentedKalmanFilterAlgorithmforRNNs 198
15.4.1 EKFBasedTrainingofComplexRNNs 200
15.5 AugmentedComplexUnscentedKalmanFilter(ACUKF) 200
15.5.1 StateSpaceEquationsfortheComplexUnscentedKalman
Filter 201
15.5.2 ACUKFBasedTrainingofComplexRNNs 202
15.6 SimulationExamples 203
16 CollaborativeAdaptiveFiltering 207
16.1 ParametricSignalModalityCharacterisation 207
16.2 StandardHybridFilteringinR 209
16.3 TrackingtheLinear/NonlinearNatureofComplexValuedSignals 210
16.3.1 SignalModalityCharacterisationinC 211
16.4 SplitvsFullyComplexSignalNatures 214
16.5 OnlineAssessmentoftheNatureofWindSignal 216
16.5.1 EffectsofAveragingonSignalNonlinearity 216
16.6 CollaborativeFiltersforGeneralComplexSignals 217
16.6.1 HybridFiltersforNoncircularSignals 218
16.6.2 OnlineTestforComplexCircularity 220
17 AdaptiveFilteringBasedonEMD 221
17.1 TheEmpiricalModeDecompositionAlgorithm 222
17.1.1 EmpiricalModeDecompositionasaFixedPointIteration 223
17.1.2 ApplicationsofRealValuedEMD 224
17.1.3 UniquenessoftheDecomposition 225
17.2 ComplexExtensionsofEmpiricalModeDecomposition 226
17.2.1 ComplexEmpiricalModeDecomposition 227
17.2.2 RotationInvariantEmpiricalModeDecomposition(RIEMD) 228
17.2.3 BivariateEmpiricalModeDecomposition(BEMD) 228
17.3 AddressingtheProblemofUniqueness 230
17.4 ApplicationsofComplexExtensionsofEMD 230
18 ValidationofComplexRepresentations–IsThisWorthwhile? 233
18.1 SignalModalityCharacterisationinR 234
18.1.1 SurrogateDataMethods 235
18.1.2 TestStatistics:TheDVVMethod 237
18.2 TestingfortheValidityofComplexRepresentation 239
18.2.1 ComplexDelayVectorVarianceMethod(CDVV) 240
18.3 QuantifyingBenefitsofComplexValuedRepresentation 243
18.3.1 ProsandConsoftheComplexDVVMethod 244
AppendixA:SomeDistinctivePropertiesofCalculusinC 245
AppendixB:Liouville’sTheorem 251
AppendixC:HypercomplexandCliffordAlgebras 253
C.1 DefinitionsofAlgebraicNotionsofGroup,RingandField 253
C.2 DefinitionofaVectorSpace 254
C.3 HigherDimensionAlgebras 254
C.4 TheAlgebraofQuaternions 255
C.5 CliffordAlgebras 256
AppendixD:RealValuedActivationFunctions 257
D.1 LogisticSigmoidActivationFunction 257
D.2 HyperbolicTangentActivationFunction 258
AppendixE:ElementaryTranscendentalFunctions(ETF) 259
AppendixF:TheONotationandStandardVectorandMatrixDifferentiation 263
F.1 TheONotation 263
F.2 StandardVectorandMatrixDifferentiation 263
AppendixG:NotionsFromLearningTheory 265
G.1 TypesofLearning 266
G.2 TheBias–VarianceDilemma 266
G.3 RecursiveandIterativeGradientEstimationTechniques 267
G.4 TransformationofInputData 267
AppendixH:NotionsfromApproximationTheory 269
AppendixI:TerminologyUsedintheFieldofNeuralNetworks 273
AppendixJ:ComplexValuedPipelinedRecurrentNeuralNetwork(CPRNN) 275
J.1 TheComplexRTRLAlgorithm(CRTRL)forCPRNN 275
J.1.1 LinearSubsectionWithinthePRNN 277
AppendixK:GradientAdaptiveStepSize(GASS)AlgorithmsinR 279
K.1 GradientAdaptiveStepsizeAlgorithmsBasedon∂J/∂μ 280
K.2 VariableStepsizeAlgorithmsBasedon∂J/∂ε 281
AppendixL:DerivationofPartialDerivativesfromChapter8 283
L.1 Derivationof∂e(k)/∂w (k) 283
n
L.2 Derivationof∂e∗(k)/∂ε(k−1) 284
L.3 Derivationof∂w(k)/∂ε(k−1) 286
AppendixM:APosterioriLearning 287
M.1 APosterioriStrategiesinAdaptiveLearning 288
AppendixN:NotionsfromStabilityTheory 291
AppendixO:LinearRelaxation 293
O.1 VectorandMatrixNorms 293
O.2 RelaxationinLinearSystems 294
O.2.1 ConvergenceintheNormorStateSpace? 297
AppendixP:ContractionMappings,FixedPointIterationandFractals 299
P.1 HistoricalPerspective 303
P.2 MoreonConvergence:ModifiedContractionMapping 305
P.3 FractalsandMandelbrotSet 308
References 309
Index 321
Preface
This book was written in response to the growing demand for a text that provides a unified
treatment of complex valued adaptive filters, both linear and nonlinear, and methods for the
processingofbothcomplexcircularandcomplexnoncircularsignals.Webelievethatthisis
thefirstattempttobringtogetherestablishedadaptivefilteringalgorithmsinCandtherecent
developmentsinthestatisticsofcomplexvariableundertheumbrellaofpowerfulmathematical
frameworks of CR (Wirtinger) calculus and augmented complex statistics. Combining the
resultsfromtheauthors’originalresearchandcurrentestablishedmethods,thisbooksserves
asarigorousaccountofexistingandnovelcomplexsignalprocessingmethods,andprovides
next generation solutions for adaptive filtering of the generality of complex valued signals.
Theintroductorychapterscanbeusedasatextforacourseonadaptivefiltering.Itisourhope
thatpeopleasexcitedaswearebythepossibilitiesopenedbythemoreadvancedworkinthis
bookwillfurtherdeveloptheseideasintonewandusefulapplications.
The title reflects our ambition to write a book which addresses several major problems
in modern complex adaptive filtering. Real world data are non-Gaussian, nonstationary and
generatedbynonlinearsystemswithpossiblylongimpulseresponses.Fortheprocessingof
such signals we therefore need nonlinear architectures to deal with nonlinearity and non-
Gaussianity, feedback to deal with long responses, and adaptive mode of operation to deal
withthenonstationarynatureofthedata.Thesehaveallbeenbroughttogetherinthisbook,
hencethetitle“ComplexValuedNonlinearAdaptiveFilters”.Thesubtitlereflectssomemore
intricateaspectsoftheprocessingofcomplexrandomvariables,andthattheclassofnonlinear
filters addressed in this work can be viewed as temporal neural networks. This material can
also be used to supplement courses on neural networks, as the algorithms developed can be
usedtotrainneuralnetworksforpatternprocessingandclassification.
Complex valued signals play a pivotal role in communications, array signal processing,
power,environmental,andbiomedicalsignalprocessingandrelatedfields.Thesesignalsare
either complex by design, such as symbols used in data communications (e.g. quadrature
phase shift keying), or are made complex by convenience of representation. The latter class
includesanalyticsignalsandsignalscomingfrommanyimportantmodernapplicationsinmag-
neticsourceimaging,interferometricradar,directionofarrivalestimationandsmartantennas,
mathematicalbiosciences,mobilecommunications,opticsandseismics.Existingbooksdonot
takeintoaccounttheeffectsonperformanceofauniquepropertyofcomplexstatistics–com-
plexnoncircularity,andemployseveralconvenientmathematicalshortcutsinthetreatmentof
complexrandomvariables.
Adaptive filters based on widely linear models introduced in this work are derived rigor-
ously,andaresuitedfortheprocessingofamuchwiderclassofcomplexnoncircularsignals
(directional processes, vector fields), and offer a number of theoretical performance gains.
Perhaps the first time we became involved in practical applications of complex adaptive fil-
tering was when trying to perform short term wind forecasting by treating wind speed and
direction,whichareroutinelyprocessedseparately,asauniquecomplexvaluedquantity.Our
resultsoutperformedthestandardapproaches.Thisopenedacanofworms,asitbecameap-
parentthatthestandardtechniqueswerenotadequate,andthatmathematicalfoundationsand
practicaltoolsfortheapplicationsofcomplexvaluedadaptivefilterstothegeneralityofcom-
plex signals are scattered throughout the literature. For instance, an often confusing aspect
of complex adaptive filtering is that the cost (objective) function to be minimised is a real
function (measure of error power) of complex variables, and is not analytic. Thus, standard
complexdifferentiability(Cauchy-Riemannconditions)doesnotapply,andweneedtoresort
topseudoderivatives.Weidentifiedtheneedforarigorous,concise,andunifiedtreatmentof
thestatisticsofcomplexvariables,methodsfordealingwithnonlinearityandnoncircularity,
andenhancedsolutionsforadaptivesignalprocessinginC,andwereencouragedbyourseries
editorSimonHaykinandthestafffromWileyChichestertoproducethistext.
The first two chapters give the introduction to the field and illustrate the benefits of the
processing in the complex domain. Chapter 1 provides a personal view of the history of
complex numbers. They are truly fascinating and, unlike other number systems which were
introducedassolutionstopracticalproblems,theyaroseasaproductofintellectualexercise.
Complex numbers were formalised in the mid-19th century by Gauss and Euler in order to
provide solutions for the fundamental theorem of algebra; within 50 years (and without the
Internet) they became a linchpin of electromagnetic field and relativity theory. Chapter 2
offerstheoreticalandpracticaljustificationforconvertingmanyapparentlyrealvaluedsignal
processingproblemsintothecomplexdomain,wheretheycanbenefitfromtheconvenienceof
representationandthepowerandbeautyofcomplexcalculus.Itillustratesthedualitybetween
theprocessinginR2andC,andthebenefitsofcomplexvaluedprocessing–unlikeR2thefield
ofcomplexnumbersformsadivisionalgebraandprovidesarigorousmathematicsframework
forthetreatmentofphase,nonlinearityandcouplingbetweensignalcomponents.
The foundations of standard complex adaptive filtering are established in Chapters 3–7.
Chapter 3 provides an overview of adaptive filtering architectures, and introduces the back-
groundfortheirstatespacerepresentationsandlinkswithpolynomialfiltersandneuralnet-
works.Chapter4dealswiththechoiceofcomplexnonlinearactivationfunctionandaddresses
thetradeoffbetweentheirboundednessandanalyticity.Theonlycontinuouslydifferentiable
functioninCthatsatisfiestheCauchy-Riemannconditionsisaconstant;topreservebound-
ednesssomeadhocapproaches(alsocalledsplit-complex)employrealvaluednonlinearities
on the real and imaginary parts. Our main interest is in complex functions of complex vari-
ables(alsocalledfullycomplex)whicharenotboundedonthewholecomplexplane,butare
complexdifferentiableandprovidesolutionswhicharegenericextensionsofthecorrespond-
ing solutions in R. Chapter 5 addresses the duality between gradient calculation in R2 and
C and introduces the so called CR calculus which is suitable for general functions of com-
plex variables, both holomorphic and non-holomorphic. This provides a unified framework
forcomputingtheJacobians,Hessians,andgradientsofcostfunctions,andservesasabasis
for the derivation of learning algorithms throughout this book. Chapters 6 and 7 introduce
standard complex valued adaptive filters, both linear and nonlinear; they are supported by
rigorousproofsofconvergence,andcanbeusedtoteachacourseonadaptivefiltering.The
complexleastmeansquare(CLMS)inChapter6isderivedstepbystep,whereasthelearning
algorithmsforfeedbackstructuresinChapter7arederivedinacompactway,basedonCR