Table Of Content(cid:2)
NONLINEAR
REGRESSION
MODELING FOR
ENGINEERING
APPLICATIONS
(cid:2) (cid:2)
(cid:2)
(cid:2)
Wiley-ASMEPressSeriesList
IntroductiontoDynamicsandControlofMechanical To March2016
EngineeringSystems
FundamentalsofMechanicalVibrations Cai May2016
NonlinearRegressionModelingforEngineering Rhinehart August2016
Applications
StressinASMEPressureVessels Jawad November2016
BioprocessingPipingandEquipmentDesign Huitt November2016
CombinedCooling,Heating,andPowerSystems Shi January2017
(cid:2) (cid:2)
(cid:2)
(cid:2)
NONLINEAR
REGRESSION
MODELING FOR
ENGINEERING
APPLICATIONS
MODELING, MODEL VALIDATION,
AND ENABLING DESIGN OF
EXPERIMENTS
(cid:2) R.RussellRhinehart (cid:2)
(cid:2)
(cid:2)
Thiseditionfirstpublished2016
©2016,JohnWiley&Sons,Ltd
FirstEditionpublishedin2016
Registeredoffice
JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UnitedKingdom
Fordetailsofourglobaleditorialoffices,forcustomerservicesandforinformationabouthowtoapplyfor
permissiontoreusethecopyrightmaterialinthisbookpleaseseeourwebsiteatwww.wiley.com.
TherightoftheauthortobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewiththeCopyright,
DesignsandPatentsAct1988.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,inany
formorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptaspermittedbytheUK
Copyright,DesignsandPatentsAct1988,withoutthepriorpermissionofthepublisher.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmaynotbe
availableinelectronicbooks.
Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks.Allbrandnamesand
productnamesusedinthisbookaretradenames,servicemarks,trademarksorregisteredtrademarksoftheir
respectiveowners.Thepublisherisnotassociatedwithanyproductorvendormentionedinthisbook
LimitofLiability/DisclaimerofWarranty:Whilethepublisherandauthorhaveusedtheirbesteffortsinpreparing
thisbook,theymakenorepresentationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsof
thisbookandspecificallydisclaimanyimpliedwarrantiesofmerchantabilityorfitnessforaparticularpurpose.Itis
soldontheunderstandingthatthepublisherisnotengagedinrenderingprofessionalservicesandneitherthe
publishernortheauthorshallbeliablefordamagesarisingherefrom.Ifprofessionaladviceorotherexpert
assistanceisrequired,theservicesofacompetentprofessionalshouldbesought.
LibraryofCongressCataloging-in-PublicationData:
Names:Rhinehart,R.Russell,1946-author.
Title:Nonlinearregressionmodelingforengineeringapplications:modeling,
(cid:2) (cid:2)
modelvalidation,andenablingdesignofexperiments/R.Russell
Rhinehart.
Description:Chichester,UK;Hoboken,NJ:JohnWiley&Sons,2016.|
Includesbibliographicalreferencesandindex.
Identifiers:LCCN2016012932(print)|LCCN2016020558(ebook)|ISBN
9781118597965(cloth)|ISBN9781118597934(pdf)|ISBN9781118597958
(epub)
Subjects:LCSH:Regressionanalysis–Mathematicalmodels.|
Engineering–Mathematicalmodels.
Classification:LCCTA342.R4952016(print)|LCCTA342(ebook)|DDC
620.001/519536–dc23
LCrecordavailableathttps://lccn.loc.gov/2016012932
AcataloguerecordforthisbookisavailablefromtheBritishLibrary.
Setin10/12pt,TimesLTStdbySPiGlobal,Chennai,India.
1 2016
(cid:2)
(cid:2)
Contents
SeriesPreface xiii
Preface xv
Acknowledgments xxiii
Nomenclature xxv
Symbols xxxvii
PartI INTRODUCTION
(cid:2) 1 IntroductoryConcepts 3 (cid:2)
1.1 IllustrativeExample – TraditionalLinearLeast-SquaresRegression 3
1.2 HowModelsAreUsed 7
1.3 NonlinearRegression 7
1.4 VariableTypes 8
1.5 Simulation 12
1.6 Issues 13
1.7 Takeaway 15
Exercises 15
2 ModelTypes 16
2.1 ModelTerminology 16
2.2 AClassificationofMathematicalModelTypes 17
2.3 Steady-StateandDynamicModels 21
2.3.1 Steady-StateModels 22
2.3.2 DynamicModels(Time-Dependent,Transient) 24
2.4 Pseudo-FirstPrinciples – AppropriatedFirstPrinciples 26
2.5 Pseudo-FirstPrinciples – Pseudo-Components 28
2.6 EmpiricalModelswithTheoreticalGrounding 28
2.6.1 EmpiricalSteadyState 28
2.6.2 EmpiricalTime-Dependent 30
2.7 EmpiricalModelswithNoTheoreticalGrounding 31
2.8 PartitionedModels 31
2.9 EmpiricalorPhenomenological? 32
(cid:2)
(cid:2)
vi Contents
2.10 EnsembleModels 32
2.11 Simulators 33
2.12 StochasticandProbabilisticModels 33
2.13 Linearity 34
2.14 DiscreteorContinuous 36
2.15 Constraints 36
2.16 ModelDesign(Architecture,Functionality,Structure) 37
2.17 Takeaway 37
Exercises 37
PartII PREPARATIONFORUNDERLYINGSKILLS
3 PropagationofUncertainty 43
3.1 Introduction 43
3.2 SourcesofErrorandUncertainty 44
3.2.1 Estimation 45
3.2.2 Discrimination 45
3.2.3 CalibrationDrift 45
3.2.4 Accuracy 45
3.2.5 Technique 46
3.2.6 ConstantsandData 46
(cid:2) 3.2.7 Noise 46 (cid:2)
3.2.8 ModelandEquations 46
3.2.9 Humans 47
3.3 SignificantDigits 47
3.4 RoundingOff 48
3.5 EstimatingUncertaintyonValues 49
3.5.1 Caution 50
3.6 PropagationofUncertainty – Overview – TwoTypes,TwoWaysEach 51
3.6.1 MaximumUncertainty 51
3.6.2 ProbableUncertainty 56
3.6.3 Generality 58
3.7 WhichtoReport?MaximumorProbableUncertainty 59
3.8 Bootstrapping 59
3.9 BiasandPrecision 61
3.10 Takeaway 65
Exercises 66
4 EssentialProbabilityandStatistics 67
4.1 VariationandItsRoleinTopics 67
4.2 HistogramandItsPDFandCDFViews 67
4.3 ConstructingaData-BasedViewofPDFandCDF 70
4.4 ParametersthatCharacterizetheDistribution 71
4.5 SomeRepresentativeDistributions 72
4.5.1 GaussianDistribution 72
(cid:2)
(cid:2)
Contents vii
4.5.2 Log-NormalDistribution 72
4.5.3 LogisticDistribution 74
4.5.4 ExponentialDistribution 74
4.5.5 BinomialDistribution 75
4.6 ConfidenceInterval 76
4.7 CentralLimitTheorem 77
4.8 HypothesisandTesting 78
4.9 TypeIandTypeIIErrors,AlphaandBeta 80
4.10 EssentialStatisticsforThisText 82
4.10.1 t-TestforBias 83
4.10.2 WilcoxonSignedRankTestforBias 83
4.10.3 r-lag-1AutocorrelationTest 84
4.10.4 RunsTest 87
4.10.5 TestforSteadyStateinaNoisySignal 87
4.10.6 Chi-SquareContingencyTest 89
4.10.7 Kolmogorov–SmirnovDistributionTest 89
4.10.8 TestforProportion 90
4.10.9 F-TestforEqualVariance 90
4.11 Takeaway 91
Exercises 91
5 Simulation 93
(cid:2) 5.1 Introduction 93 (cid:2)
5.2 ThreeSourcesofDeviation:Measurement,Inputs,Coefficients 93
5.3 TwoTypesofPerturbations:Noise(Independent)andDrifts(Persistence) 95
5.4 TwoTypesofInfluence:AdditiveandScaledwithLevel 98
5.5 UsingtheInverseCDFtoGeneratenandufromUID(0,1) 99
5.6 Takeaway 100
Exercises 100
6 SteadyandTransientStateDetection 101
6.1 Introduction 101
6.1.1 GeneralApplications 101
6.1.2 ConceptsandIssuesinDetectingSteadyState 104
6.1.3 ApproachesandIssuestoSSIDandTSID 104
6.2 Method 106
6.2.1 ConceptualModel 106
6.2.2 Equations 107
6.2.3 Coefficient,Threshold,andSampleFrequencyValues 108
6.2.4 NoiselessData 111
6.3 Applications 112
6.3.1 ApplicationsoftheR-StatisticApproachforProcessMonitoring 112
6.3.2 ApplicationsoftheR-StatisticApproachforDeterminingRegression
Convergence 112
6.4 Takeaway 114
Exercises 114
(cid:2)
(cid:2)
viii Contents
PartIII REGRESSION,VALIDATION,DESIGN
7 RegressionTarget – ObjectiveFunction 119
7.1 Introduction 119
7.2 ExperimentalandMeasurementUncertainty – StaticandContinuousValued 119
7.3 Likelihood 122
7.4 MaximumLikelihood 124
7.5 Estimatingσ andσ Values 127
x y
7.6 VerticalSSD – ALimitingConsiderationofVariabilityOnlyintheResponse
Measurement 127
7.7 r-SquareasaMeasureofFit 128
7.8 Normal,Total,orPerpendicularSSD 130
7.9 Akaho’sMethod 132
7.10 UsingaModelInverseforRegression 134
7.11 ChoosingtheDependentVariable 135
7.12 ModelPredictionwithDynamicModels 136
7.13 ModelPredictionwithClassificationModels 137
7.14 ModelPredictionwithRankModels 138
7.15 ProbabilisticModels 139
7.16 StochasticModels 139
7.17 Takeaway 139
Exercises 140
(cid:2) (cid:2)
8 Constraints 141
8.1 Introduction 141
8.2 ConstraintTypes 141
8.3 ExpressingHardConstraintsintheOptimizationStatement 142
8.4 ExpressingSoftConstraintsintheOptimizationStatement 143
8.5 EqualityConstraints 147
8.6 Takeaway 148
Exercises 148
9 TheDistortionofLinearizingTransforms 149
9.1 LinearizingCoefficientExpressioninNonlinearFunctions 149
9.2 TheAssociatedDistortion 151
9.3 SequentialCoefficientEvaluation 154
9.4 Takeaway 155
Exercises 155
10 OptimizationAlgorithms 157
10.1 Introduction 157
10.2 OptimizationConcepts 157
10.3 Gradient-BasedOptimization 159
10.3.1 NumericalDerivativeEvaluation 159
10.3.2 SteepestDescent – TheGradient 161
(cid:2)
(cid:2)
Contents ix
10.3.3 Cauchy’sMethod 162
10.3.4 IncrementalSteepestDescent(ISD) 163
10.3.5 Newton–Raphson(NR) 163
10.3.6 Levenberg–Marquardt(LM) 165
10.3.7 ModifiedLM 166
10.3.8 GeneralizedReducedGradient(GRG) 167
10.3.9 WorkAssessment 167
10.3.10 SuccessiveQuadratic(SQ) 167
10.3.11 Perspective 168
10.4 DirectSearchOptimizers 168
10.4.1 CyclicHeuristicDirectSearch 169
10.4.2 MultiplayerDirectSearchAlgorithms 170
10.4.3 Leapfrogging 171
10.5 Takeaway 173
11 MultipleOptima 176
11.1 Introduction 176
11.2 QuantifyingtheProbabilityofFindingtheGlobalBest 178
11.3 ApproachestoFindtheGlobalOptimum 179
11.4 Best-of-N RuleforRegressionStarts 180
11.5 InterpretingtheCDF 182
11.6 Takeaway 184
(cid:2) (cid:2)
12 RegressionConvergenceCriteria 185
12.1 Introduction 185
12.2 ConvergenceversusStopping 185
12.3 TraditionalCriteriaforClaimingConvergence 186
12.4 CombiningDVInfluenceonOF 188
12.5 UseRelativeImpactasConvergenceCriterion 189
12.6 Steady-StateConvergenceCriterion 190
12.7 NeuralNetworkValidation 197
12.8 Takeaway 198
Exercises 198
13 ModelDesign – DesiredandUndesiredModelCharacteristicsandEffects 199
13.1 Introduction 199
13.2 RedundantCoefficients 199
13.3 CoefficientCorrelation 201
13.4 AsymptoticandUncertaintyEffectsWhenModelisInverted 203
13.5 IrrelevantCoefficients 205
13.6 PolesandSignFlipsw.r.t.theDV 206
13.7 TooManyAdjustableCoefficientsorTooManyRegressors 206
13.8 IrrelevantModelCoefficients 215
13.8.1 StandardErroroftheEstimate 216
13.8.2 BackwardElimination 216
(cid:2)
(cid:2)
x Contents
13.8.3 LogicalTests 216
13.8.4 PropagationofUncertainty 216
13.8.5 Bootstrapping 217
13.9 Scale-UporScale-DownTransitiontoNewPhenomena 217
13.10 Takeaway 218
Exercises 218
14 DataPre-andPost-processing 220
14.1 Introduction 220
14.2 Pre-processingTechniques 221
14.2.1 Steady-andTransient-StateSelection 221
14.2.2 InternalConsistency 221
14.2.3 Truncation 222
14.2.4 AveragingandVoting 222
14.2.5 DataReconciliation 223
14.2.6 Real-TimeNoiseFilteringforNoiseReduction(MA,FoF,STF) 224
14.2.7 Real-TimeNoisefilteringforOutlierRemoval(MedianFilter) 227
14.2.8 Real-TimeNoiseFiltering,StatisticalProcessControl 228
14.2.9 ImputationofInputData 230
14.3 Post-processing 231
14.3.1 OutliersandRejectionCriterion 231
14.3.2 BimodalResidualDistributions 233
(cid:2) 14.3.3 ImputationofResponseData 235 (cid:2)
14.4 Takeaway 235
Exercises 235
15 IncrementalModelAdjustment 237
15.1 Introduction 237
15.2 ChoosingtheAdjustableCoefficientinPhenomenologicalModels 238
15.3 SimpleApproach 238
15.4 AnAlternateApproach 240
15.5 OtherApproaches 241
15.6 Takeaway 241
Exercises 241
16 ModelandExperimentalValidation 242
16.1 Introduction 242
16.1.1 Concepts 242
16.1.2 DeterministicModels 244
16.1.3 StochasticModels 246
16.1.4 Reality! 249
16.2 Logic-BasedValidationCriteria 250
16.3 Data-BasedValidationCriteriaandStatisticalTests 251
16.3.1 Continuous-Valued,Deterministic,SteadyState,orEnd-of-Batch 251
16.3.2 Continuous-Valued,Deterministic,Transient 263
16.3.3 Class/Discrete/Rank-Valued,Deterministic,Batch,orSteadyState 264
(cid:2)
