2 Main Layout Conventions of Matrix Calculus

考虑 (x), (y) 分别是 (n), (m) 维列向量, (A) 是 (mtimes n) 矩阵, (z) 是标量.

Numerator Layout

想象分子不变, 分母转置.

Vector by vector 符合直观. Jacobian.

[ frac{partial y}{partial x} = begin{pmatrix} frac{partial y_1}{partial x_1} & dots & frac{partial y_1}{partial x_n}\ vdots & ddots &vdots\ frac{partial y_m}{partial x_1} & dots & frac{partial y_m}{partial x_n} end{pmatrix} ]

Scalar by matrix 要做一次转置, 不舒服.

[ frac{partial z}{partial A}= begin{pmatrix} frac{partial z}{partial a_{11}} & dots & frac{partial z}{partial a_{m1}}\ vdots & ddots &vdots\ frac{partial z}{partial a_{1n}} & dots & frac{partial z}{partial a_{mn}} end{pmatrix} ]

Chain rule 符合直观.

[ frac{partial fcirc g}{partial x} = frac{partial f}{partial g}frac{partial g}{partial x} ]

Denominator Layout

想象分母不变, 分子转置.

Vector by vector 不舒服. Hessian.

[ frac{partial y}{partial x} = begin{pmatrix} frac{partial y_1}{partial x_1} & dots & frac{partial y_m}{partial x_1}\ vdots & ddots &vdots\ frac{partial y_1}{partial x_n} & dots & frac{partial y_m}{partial x_n} end{pmatrix} ]

Scalar by matrix 舒服.

[ frac{partial z}{partial A}= begin{pmatrix} frac{partial z}{partial a_{11}} & dots & frac{partial z}{partial a_{1n}}\ vdots & ddots &vdots\ frac{partial z}{partial a_{m1}} & dots & frac{partial z}{partial a_{mn}} end{pmatrix} ]

Chain rule "倒过来" 了, 不舒服.

[ frac{partial fcirc g}{partial x} = frac{partial g}{partial x}frac{partial f}{partial g} ]

混用

混用现象很常见. 比如 CS224n, 主体是采用 numerator layout, 但是 scalar by matrix 时是不转置的.