Smoothing Spline ANOVA

87 Pages·2017·0.51 MB·English

by  Nathaniel E. Helwig

Smoothing Spline ANOVA
Nathaniel E. Helwig
AssistantProfessorofPsychologyandStatistics
UniversityofMinnesota(TwinCities)
Updated04-Jan-2017
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide1
Copyright
Copyright (cid:13)c 2017 by Nathaniel E. Helwig
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide2
Outline of Notes
1) Introduction 3) Estimation & Inference:
Parametric regression Penalized least squares
Nonparametric regression Smoothing parameter selection
Smoothing splines Bayesian confidence intervals
2) Background Theory 4) SSANOVA in Practice:
Averaging operators One-way SSANOVA
Hilbert spaces Two-way SSANOVA (additive)
Reproducing kernels Two-waySSANOVA(interactive)
Forathoroughtreatmentsee:
Gu,C.(2013).SmoothingsplineANOVAmodels,2ndedition.NewYork:Springer-Verlag.
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide3
Introduction
Introduction
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide4
Introduction ParametricRegression
Parametric Regression Model: Scalar Form
The multiple linear regression model has the form
p
(cid:88)
y = bx +e
i j ij i
j=1
for i ∈ {1,...,n} where
y ∈ R is the real-valued response for the i-th observation
i
b ∈ R is the j-th predictor’s regression slope
j
x ∈ R is the j-th predictor for the i-th observation
ij
e ∼iid N(0,σ2) is Gaussian measurement error
i
Implies that (y|x ,...,x ) i∼nd N((cid:80)p bx ,σ2)
i i1 ip j=1 j ij
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide5
Introduction ParametricRegression
Parametric Regression Model: Matrix Form
The multiple linear regression model has the form
y = Xb+e
where
y = (y ,...,y )(cid:48) ∈ Rn is the n×1 response vector
1 n
X = [x ,...,x ] ∈ Rn×p is the n×p design matrix
1 p
• x =(x ,...,x )(cid:48) ∈Rn isj-thpredictorvector(n×1)
j 1j nj
b = (b ,...,b )(cid:48) ∈ Rp is p×1 vector of coefficients
1 p
e = (e ,...,e )(cid:48) ∈ Rn is the n×1 error vector
1 n
Implies that (y|x) ∼ N(Xb,σ2I )
n
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide6
Introduction ParametricRegression
Ordinary Least Squares Solution
The ordinary least squares (OLS) problem is
n
1 1 (cid:88)
min (cid:107)y−Xb(cid:107)2 ←→ min (y −yˆ)2
i i
b∈Rp n b∈Rp n
i=1
where (cid:107)·(cid:107) denotes the Euclidean norm and yˆ = (cid:80)p bx .
i j=1 j ij
The OLS solution has the form
bˆ = (X(cid:48)X)−1X(cid:48)y
ˆ
and the fitted values corresponding to b are given by
yˆ = Xbˆ = Hy
where H = X(X(cid:48)X)−1X(cid:48) is the hat matrix.
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide7
Introduction ParametricRegression
Summary of Results
Using the model assumption (y|x) ∼ N(Xb,σ2I ), we have
n
bˆ ∼ N(b,σ2(X(cid:48)X)−1)
yˆ ∼ N(Xb,σ2H)
eˆ ∼ N(0,σ2(I −H))
n
where eˆ = y−yˆ is the residual vector.
Typically σ2 is unknown, so we use the MSE σˆ2 = 1 (cid:80)n eˆ2.
n−p i=1 i
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide8
Introduction NonparametricRegression
Nonparametric Regression Model
The Gaussian nonparametric regression model has the form
y = η(x)+e
i i i
for i ∈ {1,...,n} where
y ∈ R is the real-valued response for the i-th observation
i
x ∈ Rp is the predictor vector for the i-th observation
i
η : Rp → R is an unknown smooth function
e ∼iid N(0,σ2) is Gaussian measurement error
i
Implies that (y|x ,...,x ) i∼nd N(η(x),σ2)
i i1 ip i
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide9
Introduction NonparametricRegression
Additive versus Interactive Models
Suppose that x = (x ,x ) with x ∈ X and x ∈ X .
i i1 i2 i1 1 i2 2
We could fit one of two possible models:
Additive : η(x) = η +η (x )+η (x )
i 0 1 i1 2 i2
Interaction : η(x) = η +η (x )+η (x )+η (x ,x )
i 0 1 i1 2 i2 12 i1 i2
where
η is a constant function
0
η is main effect of first predictor
1
η is main effect of second predictor
2
η is interaction effect
12
NathanielE.Helwig (UofMinnesota) SmoothingSplineANOVA Updated04-Jan-2017 : Slide10