Entropy Theory and its Application in Environmental and Water Engineering

658 Pages·2013·4.075 MB·English

by  Vijay P. Singh(auth.)

Entropy Theory and its Application in
Environmental and Water Engineering
Entropy
Theory and its
Application in
Environmental
and Water
Engineering
Vijay P. Singh
DepartmentofBiologicalandAgriculturalEngineering&
DepartmentofCivilandEnvironmentalEngineering
TexasA&MUniversity
Texas,USA
A John Wiley & Sons, Ltd., Publication
Thiseditionfirstpublished20132013byJohnWileyandSons,Ltd
Wiley-BlackwellisanimprintofJohnWiley&Sons,formedbythemergerofWiley’sglobalScientific,
TechnicalandMedicalbusinesswithBlackwellPublishing.
Registeredoffice:JohnWiley&Sons,Ltd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UK
Editorialoffices:9600GarsingtonRoad,Oxford,OX42DQ,UK
TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UK
111RiverStreet,Hoboken,NJ07030-5774,USA
Fordetailsofourglobaleditorialoffices,forcustomerservicesandforinformationabouthowtoapplyfor
permissiontoreusethecopyrightmaterialinthisbookpleaseseeourwebsiteat
www.wiley.com/wiley-blackwell.
TherightoftheauthortobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewiththeUK
Copyright,DesignsandPatentsAct1988.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,
inanyformorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptas
permittedbytheUKCopyright,DesignsandPatentsAct1988,withoutthepriorpermissionofthepublisher.
Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks.Allbrandnames
andproductnamesusedinthisbookaretradenames,servicemarks,trademarksorregisteredtrademarksof
theirrespectiveowners.Thepublisherisnotassociatedwithanyproductorvendormentionedinthisbook.
Thispublicationisdesignedtoprovideaccurateandauthoritativeinformationinregardtothesubjectmatter
covered.Itissoldontheunderstandingthatthepublisherisnotengagedinrenderingprofessionalservices.If
professionaladviceorotherexpertassistanceisrequired,theservicesofacompetentprofessionalshouldbe
sought.
LimitofLiability/DisclaimerofWarranty:Whilethepublisherandauthorhaveusedtheirbesteffortsin
preparingthisbook,theymakenorepresentationsorwarrantieswiththerespecttotheaccuracyor
completenessofthecontentsofthisbookandspecificallydisclaimanyimpliedwarrantiesofmerchantabilityor
fitnessforaparticularpurpose.Itissoldontheunderstandingthatthepublisherisnotengagedinrendering
professionalservicesandneitherthepublishernortheauthorshallbeliablefordamagesarisingherefrom.If
professionaladviceorotherexpertassistanceisrequired,theservicesofacompetentprofessionalshouldbe
sought.
LibraryofCongressCataloging-in-PublicationData
Singh,V.P.(VijayP.)
Entropytheoryanditsapplicationinenvironmentalandwaterengineering/VijayP.Singh.
pagescm
Includesbibliographicalreferencesandindexes.
ISBN978-1-119-97656-1(cloth)
1.Hydraulicengineering–Mathematics.2.Water–Thermalproperties–Mathematicalmodels.
3.Hydraulics–Mathematics.4.Maximumentropymethod–Congresses.5.Entropy.I.Title.
TC157.8.S462013
(cid:1)
627.0153673–dc23
2012028077
AcataloguerecordforthisbookisavailablefromtheBritishLibrary.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmaynotbe
availableinelectronicbooks.
Typesetin10/12ptTimes-RomanbyLaserwordsPrivateLimited,Chennai,India
FirstImpression2013
Dedicatedto
MywifeAnita,
sonVinay,
daughter-in-lawSonali
daughterArti,and
grandsonRonin
Contents
Preface,xv
Acknowledgments,xix
1 Introduction,1
1.1 Systemsandtheircharacteristics,1
1.1.1 Classesofsystems,1
1.1.2 Systemstates,1
1.1.3 Changeofstate,2
1.1.4 Thermodynamicentropy,3
1.1.5 Evolutiveconnotationofentropy,5
1.1.6 Statisticalmechanicalentropy,5
1.2 Informationalentropies,7
1.2.1 Typesofentropies,8
1.2.2 Shannonentropy,9
1.2.3 Informationgainfunction,12
1.2.4 Boltzmann,GibbsandShannonentropies,14
1.2.5 Negentropy,15
1.2.6 Exponentialentropy,16
1.2.7 Tsallisentropy,18
1.2.8 Renyientropy,19
1.3 Entropy,information,anduncertainty,21
1.3.1 Information,22
1.3.2 Uncertaintyandsurprise,24
1.4 Typesofuncertainty,25
1.5 Entropyandrelatedconcepts,27
1.5.1 Informationcontentofdata,27
1.5.2 Criteriaformodelselection,28
1.5.3 Hypothesistesting,29
1.5.4 Riskassessment,29
Questions,29
References,31
AdditionalReferences,32
viii Contents
2 EntropyTheory,33
2.1 Formulationofentropy,33
2.2 Shannonentropy,39
2.3 Connotationsofinformationandentropy,42
2.3.1 Amountofinformation,42
2.3.2 Measureofinformation,43
2.3.3 Sourceofinformation,43
2.3.4 Removalofuncertainty,44
2.3.5 Equivocation,45
2.3.6 Averageamountofinformation,45
2.3.7 Measurementsystem,46
2.3.8 Informationandorganization,46
2.4 Discreteentropy:univariatecaseandmarginalentropy,46
2.5 Discreteentropy:bivariatecase,52
2.5.1 Jointentropy,53
2.5.2 Conditionalentropy,53
2.5.3 Transinformation,57
2.6 Dimensionlessentropies,79
2.7 Bayestheorem,80
2.8 Informationalcorrelationcoefficient,88
2.9 Coefficientofnontransferredinformation,90
2.10 Discreteentropy:multidimensionalcase,92
2.11 Continuousentropy,93
2.11.1 Univariatecase,94
2.11.2 Differentialentropyofcontinuousvariables,97
2.11.3 Variabletransformationandentropy,99
2.11.4 Bivariatecase,100
2.11.5 Multivariatecase,105
2.12 Stochasticprocessesandentropy,105
2.13 Effectofproportionalclassinterval,107
2.14 Effectoftheformofprobabilitydistribution,110
2.15 Datawithzerovalues,111
2.16 Effectofmeasurementunits,113
2.17 Effectofaveragingdata,115
2.18 Effectofmeasurementerror,116
2.19 Entropyinfrequencydomain,118
2.20 Principleofmaximumentropy,118
2.21 Concentrationtheorem,119
2.22 Principleofminimumcrossentropy,122
2.23 Relationbetweenentropyanderrorprobability,123
2.24 Variousinterpretationsofentropy,125
2.24.1 Measureofrandomnessordisorder,125
2.24.2 Measureofunbiasednessorobjectivity,125
2.24.3 Measureofequality,125
2.24.4 Measureofdiversity,126
2.24.5 Measureoflackofconcentration,126
2.24.6 Measureofflexibility,126
Contents ix
2.24.7 Measureofcomplexity,126
2.24.8 Measureofdeparturefromuniformdistribution,127
2.24.9 Measureofinterdependence,127
2.24.10 Measureofdependence,128
2.24.11 Measureofinteractivity,128
2.24.12 Measureofsimilarity,129
2.24.13 Measureofredundancy,129
2.24.14 Measureoforganization,130
2.25 Relationbetweenentropyandvariance,133
2.26 Entropypower,135
2.27 Relativefrequency,135
2.28 Applicationofentropytheory,136
Questions,136
References,137
AdditionalReading,139
3 PrincipleofMaximumEntropy,142
3.1 Formulation,142
3.2 POMEformalismfordiscretevariables,145
3.3 POMEformalismforcontinuousvariables,152
3.3.1 EntropymaximizationusingthemethodofLagrangemultipliers,152
3.3.2 Directmethodforentropymaximization,157
3.4 POMEformalismfortwovariables,158
3.5 Effectofconstraintsonentropy,165
3.6 Invarianceoftotalentropy,167
Questions,168
References,170
AdditionalReading,170
4 DerivationofPome-BasedDistributions,172
4.1 Discretevariableanddiscretedistributions,172
4.1.1 ConstraintE[x]andtheMaxwell-Boltzmanndistribution,172
4.1.2 TwoconstraintsandBose-Einsteindistribution,174
4.1.3 TwoconstraintsandFermi-Diracdistribution,177
4.1.4 Intermediatestatisticsdistribution,178
4.1.5 Constraint:E[N]:Bernoullidistributionforasingletrial,179
4.1.6 Binomialdistributionforrepeatedtrials,180
4.1.7 Geometricdistribution:repeatedtrials,181
4.1.8 Negativebinomialdistribution:repeatedtrials,183
4.1.9 Constraint:E[N]=n:Poissondistribution,183
4.2 Continuousvariableandcontinuousdistributions,185
4.2.1 Finiteinterval[a,b],noconstraint,andrectangulardistribution,185
4.2.2 Finiteinterval[a,b],oneconstraintandtruncatedexponentialdistribution,186
4.2.3 Finiteinterval[0,1],twoconstraintsE[lnx]andE[ln(1−x)]andbeta
distributionoffirstkind,188
4.2.4 Semi-infiniteinterval(0,∞),oneconstraintE[x]andexponentialdistribution,191
4.2.5 Semi-infiniteinterval,twoconstraintsE[x]andE[lnx]
andgammadistribution,192
x Contents
4.2.6 Semi-infiniteinterval,twoconstraintsE[lnx]andE[ln(1+x)]andbeta
distributionofsecondkind,194
4.2.7 Infiniteinterval,twoconstraintsE[x]andE[x2]andnormaldistribution,195
4.2.8 Semi-infiniteinterval,log-transformationY =lnX,twoconstraintsE[y]andE[y2]
andlog-normaldistribution,197
4.2.9 Infiniteandsemi-infiniteintervals:constraintsanddistributions,199
Questions,203
References,208
AdditionalReading,208
5 MultivariateProbabilityDistributions,213
5.1 Multivariatenormaldistributions,213
5.1.1 Onetimelagserialdependence,213
5.1.2 Two-lagserialdependence,221
5.1.3 Multi-lagserialdependence,229
5.1.4 Noserialdependence:bivariatecase,234
5.1.5 Cross-correlationandserialdependence:bivariatecase,238
5.1.6 Multivariatecase:noserialdependence,244
5.1.7 Multi-lagserialdependence,245
5.2 Multivariateexponentialdistributions,245
5.2.1 Bivariateexponentialdistribution,245
5.2.2 Trivariateexponentialdistribution,254
5.2.3 ExtensiontoWeibulldistribution,257
5.3 Multivariatedistributionsusingtheentropy-copulamethod,258
5.3.1 Familiesofcopula,259
5.3.2 Application,260
5.4 Copulaentropy,265
Questions,266
References,267
AdditionalReading,268
6 PrincipleofMinimumCross-Entropy,270
6.1 ConceptandformulationofPOMCE,270
6.2 PropertiesofPOMCE,271
6.3 POMCEformalismfordiscretevariables,275
6.4 POMCEformulationforcontinuousvariables,279
6.5 RelationtoPOME,280
6.6 Relationtomutualinformation,281
6.7 Relationtovariationaldistance,281
6.8 Lin’sdirecteddivergencemeasure,282
6.9 Upperboundsforcross-entropy,286
Questions,287
References,288
AdditionalReading,289
7 DerivationofPOME-BasedDistributions,290
7.1 DiscretevariableandmeanE[x]asaconstraint,290
7.1.1 Uniformpriordistribution,291
7.1.2 Arithmeticpriordistribution,293
Contents xi
7.1.3 Geometricpriordistribution,294
7.1.4 Binomialpriordistribution,295
7.1.5 Generalpriordistribution,297
7.2 Discretevariabletakingonaninfinitesetofvalues,298
7.2.1 Improperpriorprobabilitydistribution,298
7.2.2 AprioriPoissonprobabilitydistribution,301
7.2.3 Apriorinegativebinomialdistribution,304
7.3 Continuousvariable:generalformulation,305
7.3.1 Uniformpriorandmeanconstraint,307
7.3.2 Exponentialpriorandmeanandmeanlogconstraints,308
Questions,308
References,309
8 ParameterEstimation,310
8.1 Ordinaryentropy-basedparameterestimationmethod,310
8.1.1 Specificationofconstraints,311
8.1.2 Derivationofentropy-baseddistribution,311
8.1.3 ConstructionofzerothLagrangemultiplier,311
8.1.4 DeterminationofLagrangemultipliers,312
8.1.5 Determinationofdistributionparameters,313
8.2 Parameter-spaceexpansionmethod,325
8.3 Contrastwithmethodofmaximumlikelihoodestimation(MLE),329
8.4 Parameterestimationbynumericalmethods,331
Questions,332
References,333
AdditionalReading,334
9 SpatialEntropy,335
9.1 Organizationofspatialdata,336
9.1.1 Distribution,density,andaggregation,337
9.2 Spatialentropystatistics,339
9.2.1 Redundancy,343
9.2.2 Informationgain,345
9.2.3 Disutilityentropy,352
9.3 Onedimensionalaggregation,353
9.4 Anotherapproachtospatialrepresentation,360
9.5 Two-dimensionalaggregation,363
9.5.1 Probabilitydensityfunctionanditsresolution,372
9.5.2 Relationbetweenspatialentropyandspatialdisutility,375
9.6 Entropymaximizationformodelingspatialphenomena,376
9.7 Clusteranalysisbyentropymaximization,380
9.8 Spatialvisualizationandmapping,384
9.9 Scaleandentropy,386
9.10 Spatialprobabilitydistributions,388
9.11 Scaling:ranksizeruleandZipf’slaw,391
9.11.1 Exponentiallaw,391
9.11.2 Log-normallaw,391
9.11.3 Powerlaw,392
xii Contents
9.11.4 Lawofproportionateeffect,392
Questions,393
References,394
FurtherReading,395
10 InverseSpatialEntropy,398
10.1 Definition,398
10.2 Principleofentropydecomposition,402
10.3 Measuresofinformationgain,405
10.3.1 Bivariatemeasures,405
10.3.2 Maprepresentation,410
10.3.3 Constructionofspatialmeasures,412
10.4 Aggregationproperties,417
10.5 Spatialinterpretations,420
10.6 Hierarchicaldecomposition,426
10.7 Comparativemeasuresofspatialdecomposition,428
Questions,433
References,435
11 EntropySpectralAnalyses,436
11.1 Characteristicsoftimeseries,436
11.1.1 Mean,437
11.1.2 Variance,438
11.1.3 Covariance,440
11.1.4 Correlation,441
11.1.5 Stationarity,443
11.2 Spectralanalysis,446
11.2.1 Fourierrepresentation,448
11.2.2 Fouriertransform,453
11.2.3 Periodogram,454
11.2.4 Power,457
11.2.5 Powerspectrum,461
11.3 Spectralanalysisusingmaximumentropy,464
11.3.1 Burgmethod,465
11.3.2 Kapur-Kesavanmethod,473
11.3.3 Maximizationofentropy,473
11.3.4 DeterminationofLagrangemultipliersλ ,476
k
11.3.5 Spectraldensity,479
11.3.6 Extrapolationofautocovariancefunctions,482
11.3.7 Entropyofpowerspectrum,482
11.4 Spectralestimationusingconfigurationalentropy,483
11.5 Spectralestimationbymutualinformationprinciple,486
References,490
AdditionalReading,490
12 MinimumCrossEntropySpectralAnalysis,492
12.1 Cross-entropy,492
12.2 Minimumcross-entropyspectralanalysis(MCESA),493
12.2.1 Powerspectrumprobabilitydensityfunction,493