Kernel smoother

A kernel smoother is a statistical technique to estimate a real valued function

f

:

R

p

→

R

{\displaystyle f:\mathbb {R} ^{p}\to \mathbb {R} }

as the weighted average of neighboring observed data. The weight is defined by the kernel, such that closer points are given higher weights. The estimated function is smooth, and the level of smoothness is set by a single parameter.

Kernel smoothing is a type of weighted moving average.

Definitions

Let

K

h

λ

(

X

0

,

X

)

{\displaystyle K_{h_{\lambda }}(X_{0},X)}

be a kernel defined by

K

h

λ

(

X

0

,

X

)

=

D

(

‖

X

−

X

0

‖

h

λ

(

X

0

)

)

{\displaystyle K_{h_{\lambda }}(X_{0},X)=D\left({\frac {\left\|X-X_{0}\right\|}{h_{\lambda }(X_{0})}}\right)}

where:

X

,

X

0

∈

R

p

{\displaystyle X,X_{0}\in \mathbb {R} ^{p}}

‖

⋅

‖

{\displaystyle \left\|\cdot \right\|}

is the Euclidean norm

h

λ

(

X

0

)

{\displaystyle h_{\lambda }(X_{0})}

is a parameter (kernel radius)

D(t) is typically a positive real valued function, whose value is decreasing (or not increasing) for the increasing distance between the X and X0.

Popular kernels used for smoothing include parabolic (Epanechnikov), tricube, and Gaussian kernels.

Let

Y

(

X

)

:

R

p

→

R

{\displaystyle Y(X):\mathbb {R} ^{p}\to \mathbb {R} }

be a continuous function of X. For each

X

0

∈

R

p

{\displaystyle X_{0}\in \mathbb {R} ^{p}}

, the Nadaraya-Watson kernel-weighted average (smooth Y(X) estimation) is defined by

Y

^

(

X

0

)

=

∑

i

=

1

N

K

h

λ

(

X

0

,

X

i

)

Y

(

X

i

)

∑

i

=

1

N

K

h

λ

(

X

0

,

X

i

)

{\displaystyle {\hat {Y}}(X_{0})={\frac {\sum \limits _{i=1}^{N}{K_{h_{\lambda }}(X_{0},X_{i})Y(X_{i})}}{\sum \limits _{i=1}^{N}{K_{h_{\lambda }}(X_{0},X_{i})}}}}

where:

N is the number of observed points

Y(Xi) are the observations at Xi points.

In the following sections, we describe some particular cases of kernel smoothers.

Gaussian kernel smoother

The Gaussian kernel is one of the most widely used kernels, and is expressed with the equation below.

K

(

x

∗

,

x

i

)

=

exp

⁡

(

−

(

x

∗

−

x

i

)

2

2

b

2

)

{\displaystyle K(x^{*},x_{i})=\exp \left(-{\frac {(x^{*}-x_{i})^{2}}{2b^{2}}}\right)}

Here, b is the length scale for the input space.