[비모수통계] 0. Non-parametric Function Estimation?

What does nonparametric mean?

We called the model as Nonparametric models that cannot be indexed by a finite dimensional parameter, but by infinite dimensional parameter.

 

For example, Gaussian mixture model is a parametric model that can be identified by a finite dimensional parameter. But, the model $X_1,\cdots,X_n \tilde P_\theta \in \mathcal{P}$ where $P_\theta$ has a density function $\theta\in \mathcal{F} = \{f|f\ge 0 , \int f=1, f \ \text{is "smooth"}\}$ is a one of nonparametric model.

 

If we deal with below dotted black points, it is difficult to capture the trend of these points properly with parametric regression model (colored line below). In this situation, we can use non-parametric models.

 

 

 

Various methods

Specifically, we can use kernel method or smoothing splines method.

In kernel method, most common technique is to use Gaussian kernel. It generates a following minimization problem.

$$L(a,b) = \sum_{i=1}^n \{Y_i - a-bX_i\}^2 K(x_0, X_i), \ \text{where} \ K(x_0,X_i) = \phi(\frac{X_i-x_0}{h})\frac{1}{h}$$

And $\phi$ indicates the pdf of standard normal distribution.

 

In smoothing splines methods, we find the minimizer of following equation

$$S(f) = \sum_{i=1}^n \{Y_i-f(X_i)\}^2 + \lambda \int (f^{\prime \prime})^2$$

By the value of $\lambda$, the degree of smoothness differs. When we have large $\lambda$ value, the shape of function more wiggles while small $\lambda$ forces to have straight line.

 

In the next part, I will describe more detailed explanations for various estimation methods.