导数
乘法法则
- \[\frac{\partial \mathbf{y}^{\mathrm{T}} \mathbf{z}}{\partial \mathbf{x}}=\frac{\partial \mathbf{y}}{\partial \mathbf{x}} \mathbf{z}+\frac{\partial \mathbf{z}}{\partial \mathbf{x}} \mathbf{y} \]
- \[\frac{\partial \mathbf{y}^{\mathrm{T}} A \mathbf{z}}{\partial \mathbf{x}}=\frac{\partial \mathbf{y}}{\partial \mathbf{x}} A \mathbf{z}+\frac{\partial \mathbf{z}}{\partial \mathbf{x}} A^{\mathrm{T}} \mathbf{y} \]
- \[\frac{\partial y \mathbf{z}}{\partial \mathbf{x}}=y \frac{\partial \mathbf{z}}{\partial \mathbf{x}}+\frac{\partial y}{\partial \mathbf{x}} \mathbf{z}^{\mathrm{T}} \]
链式法则 1. \[\frac{\partial \mathbf{z}}{\partial \mathbf{x}}=\frac{\partial \mathbf{y}}{\partial \mathbf{x}} \frac{\partial \mathbf{z}}{\partial \mathbf{y}} \] 2. \[\frac{\partial z}{\partial X_{i j}}=\operatorname{tr}\left(\left(\frac{\partial z}{\partial Y}\right)^{\mathrm{T}} \frac{\partial Y}{\partial X_{i j}}\right) \] 3. \[\frac{\partial z}{\partial X_{i j}}=\left(\frac{\partial z}{\partial \mathbf{y}}\right)^{\mathrm{T}} \frac{\partial \mathbf{y}}{\partial X_{i j}} \] 4. \[\frac{\partial \mathbf{g}}{\partial x}=\left(\frac{\partial \mathbf{g}}{\partial \mathbf{u}}\right)^{\mathrm{T}} \frac{\partial \mathbf{u}}{\partial x} \]
常见函数的导数
三个重要公式，太重要了！
\[\frac{\partial \mathbf{x}}{\partial \mathbf{x}}=I \]
\[\frac{\partial A \mathbf{x}}{\partial \mathbf{x}}=A^{\mathrm{T}} \]
\[\frac{\partial \mathbf{x}^{\mathrm{T}} A}{\partial \mathbf{x}}=A \]
按位计算的向量函数及其导数
\[z_{k}=f\left(x_{k}\right), \forall k=1, \cdots, K \]
\[\begin{aligned} \frac{\partial f(\mathbf{x})}{\partial \mathbf{x}} &=\left[\frac{\partial f\left(x_{j}\right)}{\partial x_{i}}\right]_{K \times K} \\ &=\left[\begin{array}{cccc}{f^{\prime}\left(x_{1}\right)} & {0} & {\cdots} & {0} \\ {0} & {f^{\prime}\left(x_{2}\right)} & {\cdots} & {0} \\ {\vdots} & {\vdots} & {\vdots} & {\vdots} \\ {0} & {0} & {\cdots} & {f^{\prime}\left(x_{K}\right)}\end{array}\right] \\ &=\operatorname{diag}\left(f^{\prime}(\mathbf{x})\right) \end{aligned} \]
Logistic 函数
\[\sigma(x)=\frac{1}{1+\exp (-x)} \]
\[\sigma^{\prime}(x)=\sigma(x)(1-\sigma(x)) \]
\[\sigma^{\prime}(\mathbf{x})=\operatorname{diag}(\sigma(\mathbf{x}) \odot(1-\sigma(\mathbf{x}))) \]