5. Linear Regression Model and Kernel-based Linear Regression Model

5.1 Linear Regression Model

假设：随机变量${\epsilon }_n$是满足独立同分布的均值为0、方差为${\sigma }^2$的高斯分布。随机变量$y_n=f({\mathbf{x}}_{\mathbf{n}})+{\epsilon }_n$。
$$E(y_n|{\mathbf{x}}_n)=f({\mathbf{x}}_n)+E({\epsilon }_n)=f({\mathbf{x}}_n)$$
对于任意$n$，假设条件期望满足
$$\begin{array}{rl}E(y_n|{\mathbf{x}}_n)& =w_0+w_1{x}_{n1}+\cdots +w_d{x}_{nd}\\ & =\left[\begin{array}{cccc}1& x_{n1}& \cdots & x_{nd}\end{array}\right]\left[\begin{array}{c}{w}_0\\ ⋮\\ w_d\end{array}\right]\\ & =[1,{\mathbf{x}}_n^T]\mathbf{w}={\bar{\mathbf{x}}}_n^T\mathbf{w}\end{array}$$
其中，${\mathbf{x}}_n=[x_{n1},\cdots ,x_{nd}{]}^T,{\bar{\mathbf{x}}}_n=[1,{\mathbf{x}}_n^T{]}^T,\mathbf{w}=[w_0,\cdots ,w_d{]}^T$。$w_0$为截距。
$$y_n=f({\mathbf{x}}_n)+{\epsilon }_n=E(y_n|{\mathbf{x}}_n)+{\epsilon }_n={\bar{\mathbf{x}}}_n^T\mathbf{w}$$
假设有$N$个样本$({\mathbf{x}}_1,y_1),\cdots ,({\mathbf{x}}_N,y_N)$，记为
$$X=\left[\begin{array}{c}{\mathbf{x}}_1^T\\ ⋮\\ {\mathbf{x}}_N^T\end{array}\right],\bar{X}=\left[\begin{array}{c}{\bar{\mathbf{x}}}_1^T\\ ⋮\\ {\bar{\mathbf{x}}}_N^T\end{array}\right]=\left[\begin{array}{cc}1& {\mathbf{x}}_1^T\\ ⋮& ⋮\\ 1& {\mathbf{x}}_N^T\end{array}\right],\mathbf{y}=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_N\end{array}\right]$$
线性回归模型描述的是$X$与$\mathbf{y}$之间的关系：
$$\{\begin{array}{l}{y}_1\approx {\bar{\mathbf{x}}}_1^T\mathbf{w}+{\epsilon }_1\\ ⋮\\ y_N\approx {\bar{\mathbf{x}}}_N^T\mathbf{w}+{\epsilon }_N\end{array}\Rightarrow \left[\begin{array}{c}{y}_1\\ ⋮\\ y_N\end{array}\right]\approx \left[\begin{array}{c}{\bar{\mathbf{x}}}_1^T\\ ⋮\\ {\bar{\mathbf{x}}}_N^T\end{array}\right]\mathbf{w}+\left[\begin{array}{c}{\epsilon }_1\\ ⋮\\ {ϵ}_N\end{array}\right]\Rightarrow \mathbf{y}\approx \bar{X}\mathbf{w}+\mathbf{\epsilon }$$
最小化残差平方和（residual sum of squares, RSS）来估计$\mathbf{w}$的最优值
$$\hat{\mathbf{w}}={\mathrm{argmin}}_{\mathbf{w}}||\mathbf{y}-\bar{X}\mathbf{w}|{|}^2$$

$$||\mathbf{y}-\bar{X}\mathbf{w}|{|}^2=(\mathbf{y}-\bar{X}\mathbf{w}{)}^T(\mathbf{y}-\bar{X}\mathbf{w})={\mathbf{y}}^T\mathbf{y}-2{\mathbf{w}}^T{\bar{X}}^T\mathbf{y}+{\mathbf{w}}^T{\bar{X}}^T\bar{X}\mathbf{w}$$

$$\frac{\partial }{\partial \mathbf{w}}||\mathbf{y}-\bar{X}\mathbf{w}|{|}^2=-2{\bar{X}}^T\mathbf{y}+2{\bar{X}}^T\bar{X}\mathbf{w}=0$$

正规方程（normal equation）为
$$\begin{gathered} {\bar{X}}^T\bar{X}\mathbf{w}={\bar{X}}^T\mathbf{y} \\ \begin{array}{rl}& \Rightarrow \hat{\mathbf{w}}=({\bar{X}}^T\bar{X}{)}^{-1}{\bar{X}}^T\mathbf{y}\\ & \Rightarrow \hat{\mathbf{y}}=\bar{X}\hat{\mathbf{w}}=\bar{X}({\bar{X}}^T\bar{X}{)}^{-1}{\bar{X}}^T\mathbf{y}\end{array} \end{gathered}$$
对于未知样本$\mathbf{x}$，$f(\mathbf{x})=[1,{\mathbf{x}}^T]\hat{\mathbf{w}}$。

$$\begin{array}{rl}\mathbf{y}\approx \bar{X}\mathbf{w}& =\left[\begin{array}{cccc}1& x_{11}& \cdots & x_{1d}\\ ⋮& ⋮& \ddots & ⋮\\ 1& x_{N1}& \cdots & x_{Nd}\end{array}\right]\left[\begin{array}{c}{w}_0\\ w_1\\ ⋮\\ w_d\end{array}\right]\\ & =w_0\left[\begin{array}{c}1\\ ⋮\\ 1\end{array}\right]+w_1\left[\begin{array}{c}{x}_{11}\\ ⋮\\ x_{N1}\end{array}\right]+\cdots +w_d\left[\begin{array}{c}{x}_{1d}\\ ⋮\\ x_{Nd}\end{array}\right]\end{array}$$

可知$\mathbf{y}$的近似值可用$\bar{X}$的列向量线性组合表示。而残差可表示为两个向量之间的差，当残差垂直于$\bar{X}$的列向量构成的线性空间时，近似值最优。

1. $$0=[1,\cdots ,1]\mathbf{e}=[1,\cdots ,1](\mathbf{y}-\hat{\mathbf{y}})=\sum _{n=1}^N{y}_n-\sum _{n=1}^N{\hat{y}}_n\Rightarrow \sum _{n=1}^N{y}_n=\sum _{n=1}^N{\hat{y}}_n\Rightarrow \bar{y}=\bar{\hat{y}}$$

2. 计算原方差
   $$\begin{array}{rl}||\mathbf{y}-\bar{y}{1}_{N\times 1}|{|}^2& =||\hat{\mathbf{y}}-\bar{y}{1}_{N\times 1}+\mathbf{y}-\hat{\mathbf{y}}|{|}^2\\ & =||\hat{\mathbf{y}}-\bar{y}{1}_{N\times 1}|{|}^2+||\mathbf{y}-\hat{\mathbf{y}}|{|}^2-2(\hat{\mathbf{y}}-\bar{y}{1}_{N\times 1}{)}^T(\mathbf{y}-\hat{\mathbf{y}})\\ & =||\hat{\mathbf{y}}-\bar{y}{1}_{N\times 1}|{|}^2+||\mathbf{y}-\hat{\mathbf{y}}|{|}^2-2(\hat{\mathbf{y}}-\bar{y}{1}_{N\times 1}{)}^T\mathbf{e}\\ & =||\hat{\mathbf{y}}-\bar{\hat{y}}{1}_{N\times 1}|{|}^2+||\mathbf{y}-\hat{\mathbf{y}}|{|}^2\end{array}$$
   估计的方差与原方差之间相差残差的平方和。

$$\begin{gathered} \sum _{n=1}^N(y_n-{\bar{y}}_n{)}^2=\sum _{n=1}^N({\hat{y}}_n-{\bar{\hat{y}}}_n{)}^2+\sum _{n=1}^N(y_n-{\hat{y}}_n{)}^2 \\ TSS=ESS+RSS \end{gathered}$$

• TSS: Total sum of squares
• ESS: Explained sum of squares
• RSS: Residual (unexplained) sum of squares

ESS与TSS的比值被用于评价拟合效果的好坏：
$$R^2=\frac{ESS}{TSS}=\frac{{\sum }_{n=1}^N({\hat{y}}_n-{\bar{\hat{y}}}_n{)}^2}{{\sum }_{n=1}^N(y_n-{\bar{y}}_n{)}^2}=\frac{\frac{1}{N}{\sum }_{n=1}^N({\hat{y}}_n-{\bar{\hat{y}}}_n{)}^2}{\frac{1}{N}{\sum }_{n=1}^N(y_n-{\bar{y}}_n{)}^2}=\frac{TSS-RSS}{TSS}=1-\frac{RSS}{TSS}$$

5.2 Kernel-based Linear Regression

$$X=\left[\begin{array}{c}\mathbf{\varphi }(x_1^T)\\ ⋮\\ \mathbf{\varphi }(x_N^T)\end{array}\right],\bar{X}=\left[\begin{array}{cc}1& \varphi ({\mathbf{x}}_1^T)\\ ⋮& ⋮\\ 1& \varphi ({\mathbf{x}}_N^T)\end{array}\right]=[1_{N\times 1},X],\mathbf{y}=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_N\end{array}\right]$$

$$\hat{\mathbf{w}}=({\bar{X}}^T\bar{X}{)}^{-1}{\bar{X}}^T\mathbf{y}=({\bar{X}}^T\bar{X})({\bar{X}}^T\bar{X}{)}^{-2}{\bar{X}}^T\mathbf{y}={\bar{X}}^T\hat{\mathbf{\alpha }},其中\hat{\mathbf{\alpha }}=\bar{X}({\bar{X}}^T\bar{X}{)}^{-2}{\bar{X}}^T\mathbf{y}$$

代入正规方程中
$$\begin{array}{rl}& {\bar{X}}^T\bar{X}\mathbf{w}={\bar{X}}^T\mathbf{y}\\ \Rightarrow & \bar{X}{\bar{X}}^T\bar{X}{\bar{X}}^T\hat{\mathbf{\alpha }}=\bar{X}{\bar{X}}^T\mathbf{y}\\ \Rightarrow & {\bar{K}}^2\hat{\mathbf{\alpha }}=\bar{K}\mathbf{y},where\bar{K}=\bar{X}{\bar{X}}^T\\ \Rightarrow & \bar{K}\hat{\mathbf{\alpha }}=\mathbf{y},ifdet(\bar{K})\ne 0\\ \Rightarrow & \hat{\mathbf{\alpha }}={\bar{K}}^{-1}\mathbf{y}\end{array}$$

$$\bar{K}=\bar{X}{\bar{X}}^T=[1_{N\times 1},X]\left[\begin{array}{c}{1}_{1\times N}\\ X^T\end{array}\right]=1_{N\times 1}+X{X}^T=1_{N\times 1}+K$$

因此，
$$\begin{array}{rl}\hat{\mathbf{\alpha }}& =(1_{N\times 1}+K{)}^{-1}\mathbf{y}\\ \hat{\mathbf{y}}& =\bar{X}\mathbf{w}=\bar{X}{\bar{X}}^T\hat{\mathbf{\alpha }}=(1_{N\times 1}+K)\hat{\mathbf{\alpha }}\end{array}$$
对于未知的$\mathbf{x}$,
$$\begin{array}{rl}f(\mathbf{x})& =[1,\varphi (\mathbf{x}{)}^T]\hat{\mathbf{w}}\\ & =[1,\varphi (\mathbf{x}{)}^T]{\bar{X}}^T\hat{\mathbf{\alpha }}\\ & =[1,\varphi (\mathbf{x}{)}^T]\left[\begin{array}{c}{1}_{1\times N}\\ X^T\end{array}\right]\hat{\mathbf{\alpha }}\\ & =(1_{1\times N}+(X\varphi (\mathbf{x}){)}^T)\hat{\mathbf{\alpha }}\\ & =[1+\kappa ({\mathbf{x}}_1,\mathbf{x}),\cdots ,1+\kappa ({\mathbf{x}}_1,\mathbf{x})]\hat{\mathbf{\alpha }}\\ & =({\hat{\alpha }}_1+\cdots +{\hat{\alpha }}_N)+{\hat{\alpha }}_1\kappa ({\mathbf{x}}_1,\mathbf{x})+\cdots +{\hat{\alpha }}_N\kappa ({\mathbf{x}}_1,\mathbf{x})\end{array}$$

最后

以上就是甜蜜饼干最近收集整理的关于Kernel Method: 5. 线性回归5. Linear Regression Model and Kernel-based Linear Regression Model的全部内容，更多相关Kernel内容请搜索靠谱客的其他文章。