Non-linear least squares

Non-linear least squares is the form of least squares analysis used to fit a set of m observations with a model that is non-linear in n unknown parameters (m ≥ n). It is used in some forms of nonlinear regression. The basis of the method is to approximate the model by a linear one and to refine the parameters by successive iterations. There are many similarities to linear least squares, but also some significant differences. In economic theory, the non-linear least squares method is applied in (i) the probit regression, (ii) threshold regression, (iii) smooth regression, (iv) logistic link regression, (v) Box–Cox transformed regressors (

m

(

x

,

θ

i

)

=

θ

1

+

θ

2

x

(

θ

3

)

{\displaystyle m(x,\theta _{i})=\theta _{1}+\theta _{2}x^{(\theta _{3})}}

).

Theory

Consider a set of

m

{\displaystyle m}

data points,

(

x

1

,

y

1

)

,

(

x

2

,

y

2

)

,

…

,

(

x

m

,

y

m

)

,

{\displaystyle (x_{1},y_{1}),(x_{2},y_{2}),\dots ,(x_{m},y_{m}),}

and a curve (model function)

y

^

=

f

(

x

,

β

)

,

{\displaystyle {\hat {y}}=f(x,{\boldsymbol {\beta }}),}

that in addition to the variable

x

{\displaystyle x}

also depends on

n

{\displaystyle n}

parameters,

β

=

(

β

1

,

β

2

,

…

,

β

n

)

,

{\displaystyle {\boldsymbol {\beta }}=(\beta _{1},\beta _{2},\dots ,\beta _{n}),}

with

m

≥

n

.

{\displaystyle m\geq n.}

It is desired to find the vector

β

{\displaystyle {\boldsymbol {\beta }}}

of parameters such that the curve fits best the given data in the least squares sense, that is, the sum of squares

S

=

∑

i

=

1

m

r

i

2

{\displaystyle S=\sum _{i=1}^{m}r_{i}^{2}}

is minimized, where the residuals (in-sample prediction errors) ri are given by

r

i

=

y

i

−

f

(

x

i

,

β

)

{\displaystyle r_{i}=y_{i}-f(x_{i},{\boldsymbol {\beta }})}

for

i

=

1

,

2

,

…

,

m

.

{\displaystyle i=1,2,\dots ,m.}

The minimum value of S occurs when the gradient is zero. Since the model contains n parameters there are n gradient equations:

∂

S

∂

β

j

=

2

∑

i

r

i

∂

r

i

∂

β

j

=

0

(

j

=

1

,

…

,

n

)

.

{\displaystyle {\frac {\partial S}{\partial \beta _{j}}}=2\sum _{i}r_{i}{\frac {\partial r_{i}}{\partial \beta _{j}}}=0\quad (j=1,\ldots ,n).}

In a nonlinear system, the derivatives

∂

r

i

∂

β

j

{\textstyle {\frac {\partial r_{i}}{\partial \beta _{j}}}}

are functions of both the independent variable and the parameters, so in general these gradient equations do not have a closed solution.