Linear Regression - Normal Equation and Regularization

In linear regression problems, we can use a method called Normal Equation to fit the parameters.
Suppose we have a training set like this:
$$X=\left[\begin{array}{c}(x^{(1)}{)}^T\\ (x^{(2)}{)}^T\\ ...\\ (x^{(m)}{)}^T\end{array}\right]$$
where:
$$x^{(i)}=\left[\begin{array}{c}{x}_0^{(i)}\\ x_1^{(i)}\\ ...\\ x_n^{(i)}\end{array}\right]$$
and the label set:
$$y=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ ...\\ y^{(m)}\end{array}\right]$$
We wants to fit parameters
$$\theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ...\\ {\theta }_n\end{array}\right]$$
to make this equation:
$$J=||X\cdot \theta -y|{|}^2$$
to have its global minimum. which is:
$$\theta ={\mathrm{argmin}}_{\theta }||X\cdot \theta -y|{|}^2=(X^{T}X{)}^{-1}\cdot X^{T}y$$
Let’s prove it.
We take the partial derivatives of each parameters. for ${\theta }_j$, we find that:
$$\frac{\partial J}{\partial {\theta }_j}=\sum _{i=1}^m((x^{(i)}{)}^T\theta -y^{(i)})\cdot x_j^{(i)}=0$$
tranform this quation, we find that:
$$\left[\begin{array}{cccc}{x}_j^{(1)}& x_j^{(2)}& ...& x_j^{(m)}\end{array}\right]X\cdot \theta =\left[\begin{array}{cccc}{x}_j^{(1)}& x_j^{(2)}& ...& x_j^{(m)}\end{array}\right]\cdot y$$
combine all the n+1 equations, we find that:
$$X^{T}X\theta =X^{T}y$$
$$\theta =(X^{T}X{)}^{-1}{X}^{T}y$$

Then we involve reguarization, which means, we want to change the function J to be:
$$J=||X\cdot \theta -y|{|}^2+\lambda \sum _{j=1}^n{\theta }_j^2$$
where $\lambda$ is a constant called the regularization parameter.
Still, we calculate the partial derivative for each ${\theta }_j$. Note that the partial derivative for ${\theta }_0$ is not change.
$$\frac{\partial J}{\partial {\theta }_j}=\sum _{i=1}^m((x^{(i)}{)}^T\theta -y^{(i)})\cdot x_j^{(i)}+\lambda {\theta }_j=0(forj>0)$$
$$\left[\begin{array}{cccc}{x}_j^{(1)}& x_j^{(2)}& ...& x_j^{(m)}\end{array}\right]X\cdot \theta +\lambda {\theta }_j=\left[\begin{array}{cccc}{x}_j^{(1)}& x_j^{(2)}& ...& x_j^{(m)}\end{array}\right]\cdot y$$
$$\lambda {\theta }_j=\lambda e_j^T\theta$$
where $e_j$ is the unit vector with the jth element be 1 and others be 0
We add all the n+1 equations up, to find :
$$(X^{T}X+\lambda L)\theta =X^{T}y$$
$$\theta =(X^{T}X+\lambda L{)}^{-1}{X}^{T}y$$
where
$$L=diag(0,1,1,...,1)$$

最后

以上就是拼搏树叶最近收集整理的关于Linear Regression - Normal Equation and Regularization的全部内容，更多相关Linear内容请搜索靠谱客的其他文章。