Table Of ContentRegression
Analysis
AN INTUITIVE GUIDE
Jim Frost
Copyright © 2019 by Jim Frost.
All rights reserved. No part of this publication may be reproduced, distributed
or transmitted in any form or by any means, including photocopying, record-
ing, or other electronic or mechanical methods, without the prior written per-
mission of the publisher, except in the case of brief quotations embodied in
critical reviews and certain other noncommercial uses permitted by copyright
law.
To contact the author, please email: [email protected].
Visit the author’s website at statisticsbyjim.com.
Ordering Information:
Quantity sales. Special discounts are available on quantity purchases by edu-
cators. For details, contact the email address above.
Regression Analysis / Jim Frost. —1st ed.
3
Contents
My Approach to Teaching Regression and Statistics ...... 11
Correlation and an Introduction to Regression ............... 14
Graph Your Data to Find Correlations .......................... 15
Interpret the Pearson’s Correlation Coefficient .......... 16
Graphs for Different Correlations .................................. 17
Discussion about the Correlation Scatterplots ............. 20
Pearson’s Correlation Measures Linear Relationships 21
Hypothesis Test for Correlations ................................... 22
Interpreting our Height and Weight Example ............. 22
Correlation Does Not Imply Causation ......................... 23
How Strong of a Correlation is Considered Good? ..... 23
Common Themes with Regression ................................ 24
Regression Takes Correlation to the Next Level ......... 25
Fundamental Terms and Goals of Regression .............. 25
Regression Analyzes a Wide Variety of Relationships27
Using Regression to Control Independent Variables .. 29
An Introduction to Regression Output .......................... 30
Review and Next Steps ..................................................... 31
i
Regression Basics and How it Works ................................ 33
Data Considerations for OLS ........................................... 34
How OLS Fits the Best Line ............................................. 35
Implications of Minimizing SSE ...................................... 40
Other Types of Sums of Squares ..................................... 41
Displaying a Regression Model on a Fitted Line Plot .. 43
Importance of Staying Close to Your Data ................... 44
Review and Next Steps ..................................................... 46
Interpreting Main Effects and Significance ...................... 47
Regression Notation ......................................................... 48
Fitting Models is an Iterative Process ............................ 49
Three Types of Effects in Regression Models .............. 50
Main Effects of Continuous Variables ........................... 51
Recoding Continuous Independent Variables .............. 59
Main Effects of Categorical Variables ............................ 62
Blurring the Continuous and Categorical Line ............. 72
Constant (Y Intercept) ..................................................... 74
Review and Next Steps ..................................................... 82
Fitting Curvature .................................................................. 83
Example Curvature ........................................................... 84
Graph Curvature with Main Effects Plots ..................... 86
Why You Need to Fit Curves in a Regression Model .. 88
Difference between Linear and Nonlinear Models ...... 89
Finding the Best Way to Model Curvature ................... 93
Another Curve Fitting Example .................................... 105
Review and Next Steps ................................................... 110
Interaction Effects .............................................................. 111
Example with Categorical Independent Variables..... 112
How to Interpret Interaction Effects ........................... 113
Overlooking Interaction Effects is Dangerous! .......... 115
Example with Continuous Independent Variables .... 116
Important Considerations for Interaction Effects ..... 118
Common Questions about Interaction Effects ........... 119
Review and Next Steps ................................................... 124
Goodness-of-Fit ................................................................... 125
Assessing the Goodness-of-Fit ...................................... 125
R-squared .......................................................................... 126
Visual Representation of R-squared ............................. 127
R-squared has Limitations ............................................. 129
Are Low R-squared Values Always a Problem? ......... 129
Are High R-squared Values Always Great?................. 130
R-squared Is Not Always Straightforward .................. 132
Adjusted R-Squared and Predicted R-Squared ........... 132
iii
A Caution about Chasing a High R-squared ................ 138
Standard Error of the Regression vs. R-squared ........ 139
The F-test of Overall Significance ................................ 144
Review and Next Steps ................................................... 146
Specify Your Model ............................................................ 148
The Importance of Graphing Your Data ..................... 149
Statistical Methods for Model Specification ............... 151
Real World Complications ............................................ 153
Practical Recommendations .......................................... 154
Omitted Variable Bias ..................................................... 156
Automated Variable Selection Procedures ................. 167
Stepwise versus Best Subsets ........................................ 173
Review and Next Steps ................................................... 179
Problematic Methods of Specifying Your Model ........... 181
Using Data Dredging and Significance ......................... 182
Overfitting Regression Models ..................................... 188
Review and Next Steps ................................................... 193
Checking Assumptions and Fixing Problems ................. 195
Check Your Residual Plots! ........................................... 196
The Seven Classical OLS Assumptions ........................ 201
Heteroscedasticity .......................................................... 211
Multicollinearity .............................................................. 220
Unusual Observations ..................................................... 230
Using Data Transformations to Fix Problems ............ 242
Cheat Sheet for Detecting and Solving Problems ...... 248
Using Regression to Make Predictions ............................ 253
Explanatory versus Predictive Models ........................ 254
The Regression Approach for Predictions .................. 256
Example Scenario for Regression Predictions ............ 257
Finding a Good Regression Model for Predictions .... 257
The Illusion of Predictability ......................................... 264
Different Example of Using Prediction Intervals....... 272
Tips, Common Questions, and Concerns ........................ 277
Five Tips to Avoid Common Problems ....................... 277
Identifying the Most Important Variables .................. 282
Comparing Regression Lines with Hypothesis Tests 289
How High Does R-squared Need to Be? ...................... 294
Five Reasons Why R-squared can be Too High ......... 299
Interpreting Models that have Significant Variables but
a Low R-squared .............................................................. 304
Choosing the Correct Type of Regression ...................... 311
Continuous Dependent Variables ................................. 312
Categorical Dependent Variables ................................. 315
Count Dependent Variables .......................................... 317
v
Examples of Other Types of Regression ......................... 319
Using Log-Log Plots to Determine Whether Size Matters
............................................................................................ 319
Binary Logistic Regression: Statistical Analysis of the
Republican Establishment Split .................................... 326
References ........................................................................... 333
About the Author ................................................................ 334
