Derivatives of matrix1. matrix V scalar2. scalar V matrix3. vector V vector4. Examples5. Reference

66 阅读 0 评论 44 点赞

我是靠谱客的博主爱笑仙人掌，最近开发中收集的这篇文章主要介绍Derivatives of matrix1. matrix V scalar2. scalar V matrix3. vector V vector4. Examples5. Reference，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

matrix V scalar
scalar V matrix
vector V vector
Examples
- 1 Derivative of inverse matrix
- 2 Derivative of trace
- 3 Applications
- 4 Derivative of determinant
Reference

1. matrix V scalar

\partial A ( x ) \partial x = [\partial A ( x ) i j \partial x] i, j

$begin{equation} frac{partial A(x)}{partial x} = left[frac{partial A(x)_{ij}}{partial x}right]_{i,j} end{equation}$

2. scalar V matrix

f ( A ) \partial A = [\partial f ( x ) \partial A i j] i, j

$begin{equation} frac{f(A)}{partial A} = left[frac{partial f(x)}{partial A_{ij}}right]_{i,j} end{equation}$

3. vector V vector

g \to ( V \to ) \partial V \to = ⎡ ⎣ ⎢ g \to ( V \to ) i \partial V \to j ⎤ ⎦ ⎥ i, j

$begin{equation} frac{overrightarrow g( overrightarrow V)}{partial overrightarrow V} = left[frac{overrightarrow g( overrightarrow V)_i}{partial overrightarrow V_j}right]_{i,j} end{equation}$

4. Examples

4.1 Derivative of inverse matrix

\Rightarrow \Rightarrow \Rightarrow A - 1 A = I \partial A - 1 A \partial x = 0 \partial A - 1 \partial x A + A - 1 \partial A \partial x = 0 \partial A - 1 \partial x = - A - 1 \partial A \partial x A - 1

$begin{align} & A^{-1}A = I\ Rightarrow&frac{partial A^{-1}A }{partial x}=0\ Rightarrow&frac{partial A^{-1}}{partial x}A+A^{-1}frac{partial A}{partial x}=0\ Rightarrow& frac{partial A^{-1}}{partial x}=-A^{-1}frac{partial A}{partial x}A^{-1} end{align}$

For $x=A_{ij}$ with $A$ being nonsymmetric, we have

\partial A \partial A i j = I i j

$frac{partial A}{partial A_{ij}}=I_{ij}$
and

\partial A - 1 \partial A i j = - A - 1 I i j A - 1

$frac{partial A^{-1}}{partial A_{ij}} =-A^{-1}I_{ij}A^{-1}$
and
$\partial 2 A - 1 \partial A i j \partial A k r = \partial \partial A - 1 \partial A k r \partial A i j = - \partial A - 1 I k r A - 1 \partial A i j = - \partial A - 1 \partial A i j I k r A - 1 - A - 1 I k r \partial A - 1 I k r A - 1 \partial A i j = - (- A - 1 I i j A - 1) I k r A - 1 - A - 1 I k r (- A - 1 I i j A - 1) I k r = A - 1 I i j A - 1 I k r A - 1 + A - 1 I k r A - 1 I i j A - 1$ $begin{align} frac{partial^2 A^{-1}}{partial A_{ij}partial A_{kr}} &=frac{partial frac{partial A^{-1}}{partial A_{kr}}}{partial A_{ij}} =frac{-partial A^{-1}I_{kr}A^{-1}}{partial A_{ij}}\ &=- frac{partial A^{-1}}{partial A_{ij}}I_{kr}A^{-1} - A^{-1}I_{kr}frac{partial A^{-1}I_{kr}A^{-1}}{partial A_{ij}}\ &=-left(-A^{-1}I_{ij}A^{-1}right)I_{kr}A^{-1} - A^{-1}I_{kr}left(-A^{-1}I_{ij}A^{-1}right)I_{kr}\ &=A^{-1}I_{ij}A^{-1}I_{kr}A^{-1} +A^{-1}I_{kr}A^{-1}I_{ij}A^{-1} end{align}$

For $x=A_{ij}$ with $A$ being symmetric, we have

$\partial A \partial A i j = I i j + I j i - δ j i I j i = d I * i j$ $frac{partial A}{partial A_{ij}}=I_{ij}+I_{ji} - delta_{ji}I_{ji}overset{d}{=}I_{ij}^*$

Then all can be replaced by $I_{ij}^*$

4.2 Derivative of trace

If A is not symmetric:

$\partial tr ( A ) \partial A = [\partial tr ( A ) \partial A i j] i j = [tr (\partial A \partial A i j)] i j = tr [I i j] i j = I n$ $begin{align} frac{partial text{tr}(A)}{partial A} = left[frac{partial text{tr}(A)}{partial A_{ij}}right]_{ij} = left[text{tr}left(frac{partial A}{partial A_{ij}}right)right]_{ij} =text{tr}[I_{ij}]_{ij} = I_n end{align}$
If A is symmetric:
$\partial tr ( A ) \partial A = [\partial tr ( A ) \partial A i j] i j = [tr (\partial A \partial A i j)] i j = tr [I i j + I j i + δ j i I j i] i j = I n$ $begin{align} frac{partial text{tr}(A)}{partial A} = left[frac{partial text{tr}(A)}{partial A_{ij}}right]_{ij} = left[text{tr}left(frac{partial A}{partial A_{ij}}right)right]_{ij} = text{tr}left[I_{ij} + I_{ji}+delta_{ji}I_{ji}right]_{ij} = I_{n} end{align}$

If S is not symmetric:

$\Rightarrow \Rightarrow f = tr (S - 1 B) \partial f \partial S i j = \partial tr ( S - 1 B ) \partial S i j = tr (\partial S - 1 B S i j) = - tr (S - 1 I i j S - 1 B) = tr (I i j S - 1 B S - 1) =    Θ = S - 1 B S - 1 tr (I i j Θ) = Θ j i \partial tr ( S - 1 B ) \partial S = - (S - 1 B S - 1) T$ $begin{align} &f = text{tr}left(S^{-1}Bright)\ Rightarrow &frac{partial f}{partial S_{ij}} = frac{partial text{tr}left(S^{-1}Bright)}{partial S_{ij}}=text{tr}left(frac{partial S^{-1}B}{S_{ij}}right) = - text{tr}left(S^{-1}I_{ij}S^{-1}Bright) \ &= text{tr}left(I_{ij}S^{-1}BS^{-1}right) underbrace{=}_{Theta=S^{-1}BS^{-1}}text{tr}left(I_{ij}Thetaright) = Theta_{ji}\ Rightarrow &frac{partial text{tr}left(S^{-1}Bright)}{partial S} = - left(S^{-1}BS^{-1}right)^T end{align}$

If S is symmetric:

$\Rightarrow \Rightarrow f = tr (S - 1 B) \partial f \partial S i j = \partial tr ( S - 1 B ) \partial S i j = tr (\partial S - 1 B S i j) = - tr (I * i j Θ) = - [Θ j i + Θ i j - δ i j Θ i j] \partial tr ( S - 1 B ) \partial S = - [S - 1 (B + B T) S - 1 - diag (S - 1 B S - 1)]$ $begin{align} &f = text{tr}left(S^{-1}Bright)\ Rightarrow &frac{partial f}{partial S_{ij}} = frac{partial text{tr}left(S^{-1}Bright)}{partial S_{ij}}=text{tr}left(frac{partial S^{-1}B}{S_{ij}}right) = - text{tr}left(I_{ij}^*Thetaright) = -[Theta_{ji}+Theta_{ij} - delta_{ij}Theta_{ij}]\ Rightarrow &frac{partial text{tr}left(S^{-1}Bright)}{partial S} = - left[S^{-1}(B+B^T)S^{-1}- text{diag}left(S^{-1}BS^{-1}right)right] end{align}$

4.3 Applications

$\partial tr ( A B ) \partial A = B T$ $frac{partial text{tr}(AB)}{partial A} = B^T$
$\partial tr ( A T B ) \partial A = B$ $frac{partial text{tr}(A^TB)}{partial A} = B$

$\partial tr ( A T S A ) \partial A = [\partial tr ( A T S A ) \partial A i j] i j = [tr (I j i S A + A T S I i j)] i j = S A + S A T A = (S + S T) A$ $begin{align} frac{partial text{tr}(A^T S A)}{partial A} =left[frac{partial text{tr}(A^T S A)}{partial A_{ij}} right]_{ij} =left[text{tr}left(I_{ji}SA+A^T S I_{ij}right)right]_{ij} =SA+SA^TA = (S+S^T)A end{align}$

S is symmetric

$\partial tr ( ( A T S A ) - 1 R ) \partial A = ⎡ ⎣ ⎢ ⎢ ⎢ \partial tr ( ( A T S A ) - 1 R ) \partial A i j ⎤ ⎦ ⎥ ⎥ ⎥ i j = ⎡ ⎣ ⎢ ⎢ tr ⎛ ⎝ ⎜ ⎜ \partial ( A T S A ) - 1 R \partial A i j ⎞ ⎠ ⎟ ⎟ ⎤ ⎦ ⎥ ⎥ i j = [tr (- (A T S A) - 1 \partial A T S A \partial A i j (A T S A) - 1 R)] i j = [tr (- (A T S A) - 1 (I j i S A + A T S I i j) (A T S A) - 1 R)] i j = [- tr (I j i S A (A T S A) - 1 R (A T S A) - 1)] i j + [- tr ((A T S A) - 1 R (A T S A) - 1 A T S I i j)] i j = - S A (A T S A) - 1 R (A T S A) - 1 - S A (A T S A) - 1 R T (A T S A) - 1 = - S A (A T S A) - 1 (R + R T) (A T S A) - 1$ $begin{align} &frac{partial text{tr}left(left(A^T S Aright)^{-1}Rright)}{partial A} \ &= left[frac{partial text{tr}left(left(A^T S Aright)^{-1}Rright)}{partial A_{ij}} right]_{ij}=left[text{tr}left(frac{partial left(A^T S Aright)^{-1} R }{partial A_{ij}}right)right]_{ij} \ &=left[text{tr}left(-left(A^T S Aright)^{-1}frac{partial A^T S A}{partial A_{ij}}left(A^T S Aright)^{-1} Rright)right]_{ij}\ &=left[text{tr}left(-left(A^T S Aright)^{-1}left(I_{ji}SA +A^T S I_{ij}right)left(A^T S Aright)^{-1} Rright)right]_{ij}\ &=left[- text{tr}left(I_{ji}SAleft(A^T S Aright)^{-1} R left(A^T S Aright)^{-1}right)right]_{ij}\ & +left[- text{tr}left(left(A^T S Aright)^{-1} Rleft(A^T S Aright)^{-1}A^T S I_{ij}right)right]_{ij}\ &=-SAleft(A^T S Aright)^{-1} R left(A^T S Aright)^{-1} - SAleft(A^T S Aright)^{-1} R^T left(A^T S Aright)^{-1}\ &=-SAleft(A^T S Aright)^{-1} left(R+R^Tright) left(A^T S Aright)^{-1} end{align}$

$\partial tr ( ( A T S 2 A ) - 1 ( A T S 1 A ) ) \partial A = \partial tr ( ( A T 1 S 2 A 1 ) - 1 ( A T S 1 A ) ) \partial A ∣ ∣ ∣ ∣ ∣ ∣ A 1 = A + \partial tr ( ( A T S 2 A ) - 1 ( A T 2 S 1 A 2 ) ) \partial A ∣ ∣ ∣ ∣ ∣ ∣ A 2 = A = - 2 S 2 A (A T S 2 A) - 1 (A T S 1 A) (A T S 2 A) - 1 + 2 S 1 A (A T S 2 A) - 1$ $begin{align} &frac{partial text{tr}left( left(A^TS_2 Aright)^{-1}left(A^TS_1Aright)right)}{partial A}\ &=left.frac{partial text{tr}left( left(A_1^TS_2 A_1right)^{-1}left(A^TS_1Aright)right)}{partial A}right|_{A_1=A}\ &+left.frac{partial text{tr}left( left(A^TS_2 Aright)^{-1}left(A_2^TS_1A_2right)right)}{partial A}right|_{A_2=A}\ &=-2S_2A left(A^TS_2 Aright)^{-1}left(A^TS_1Aright)left(A^TS_2 Aright)^{-1}+2S_1Aleft(A^TS_2 Aright)^{-1} end{align}$

4.4 Derivative of determinant

$\partial | R | \partial R i j = C i j$ $frac{partial |R|}{partial R_{ij}}=C_{ij}$
which is its cofactor(from the cofactor expansion). Then
$\partial | R | \partial R = C$ $frac{partial |R|}{partial R}=C$
which is the adjoint matrix. Then from the fact that
$| R | I = R C T$ $|R|I=R C^T$
we have
$\partial | R | \partial R = | R | R - 1 T$ $frac{partial |R|}{partial R}=|R|{R^{-1}}^T$

Furthermore

$\partial ln | R | \partial R = 1 | R | \partial | R | \partial R = R - 1 T$ $frac{partial text{ln}|R|}{partial R}=frac{1}{|R|}frac{partial |R|}{partial R}={R^{-1}}^T$

5. Reference

reference of book “Introduction to Statistical Pattern Recognition, second edition.”