softmax函数：
S
i
=
e
a
i
∑
j
e
a
j
S_{i}=frac{e^{a_{i}}}{sum_{j} e^{a_{j}}}
Si=∑jeajeai
softmax求导
∂ S i ∂ a j = ∂ e a i ∂ a j ⋅ Σ − ∂ Σ ∂ a j ⋅ e a i ∑ 2 frac{partial S_{i}}{partial a_{j}}=frac{frac{partial e^{a_{i}}}{partial a_{j}} cdot Sigma-frac{partial Sigma}{partial a_{j}} cdot e^{a_{i}}}{sum^{2}} ∂aj∂Si=∑2∂aj∂eai⋅Σ−∂aj∂Σ⋅eai
当
i
=
j
i = j
i=j时：
∂
S
i
∂
a
j
=
e
a
i
⋅
Σ
−
e
a
j
e
a
i
Σ
2
=
e
a
i
Σ
⋅
Σ
−
e
a
j
Σ
=
S
i
⋅
(
1
−
S
j
)
frac{partial S_{i}}{partial a_{j}}=frac{e^{a_{i}} cdot Sigma-e^{a_{j}} e^{a_{i}}}{Sigma^{2}} = frac{e^{a_{i}}}{Sigma} cdot frac{Sigma-e^{a_{j}}}{Sigma}=S_{i} cdotleft(1-S_{j}right)
∂aj∂Si=Σ2eai⋅Σ−eajeai=Σeai⋅ΣΣ−eaj=Si⋅(1−Sj)
当
i
≠
j
i neq j
i=j时：
∂
S
i
∂
a
j
=
−
e
α
j
⋅
e
a
i
Σ
2
=
−
S
i
⋅
S
j
frac{partial S_{i}}{partial a_{j}}=-frac{e^{alpha_{j}} cdot e^{a_{i}}}{Sigma^{2}}=-S_{i} cdot S_{j}
∂aj∂Si=−Σ2eαj⋅eai=−Si⋅Sj
交叉熵损失函数
L = − ∑ y i log S i L=-sum y_{i} log S_{i} L=−∑yilogSi
∂ L ∂ S i = − y i ⋅ 1 S i frac{partial L}{partial S_{i}}=-y_{i} cdot frac{1}{S_{i}} ∂Si∂L=−yi⋅Si1
对softmax的输入求导：
∂ L ∂ a i = ∑ j ∂ L ∂ S j ⋅ ∂ S j ∂ a i = ∂ L ∂ S i ⋅ ∂ S i ∂ a i + ∑ j ≠ i ∂ L ∂ S j ⋅ ∂ S j ∂ a i = − y i ⋅ 1 S i ⋅ S i ( 1 − S i ) + ∑ j ≠ i − y j S j ⋅ ( − 1 ) S i S j = − y i ( 1 − S i ) + ∑ j ≠ i y j ⋅ S i = − y i + y i S i + ∑ j ≠ i y j ⋅ S i = S i − y i frac{partial L}{partial a_{i}}=sum_{j} frac{partial L}{partial S_{j}} cdot frac{partial S_{j}}{partial a_{i}}=frac{partial L}{partial S_{i}} cdot frac{partial S_{i}}{partial a_{i}}+sum_{j neq i} frac{partial L}{partial S_{j}} cdot frac{partial S_{j}}{partial a_{i}} \ =-y_{i} cdot frac{1}{S_{i}} cdot S_{i}left(1-S_{i}right)+sum_{j neq i} frac{-y_{j}}{S_{j}} cdot(-1) S_{i} S_{j} \ =-y_{i}left(1-S_{i}right)+sum_{j neq i} y_{j} cdot S_{i} \ =-y_{i}+y_{i} S_{i}+sum_{j neq i} y_{j} cdot S_{i} \ =S_{i}-y_{i} ∂ai∂L=j∑∂Sj∂L⋅∂ai∂Sj=∂Si∂L⋅∂ai∂Si+j=i∑∂Sj∂L⋅∂ai∂Sj=−yi⋅Si1⋅Si(1−Si)+j=i∑Sj−yj⋅(−1)SiSj=−yi(1−Si)+j=i∑yj⋅Si=−yi+yiSi+j=i∑yj⋅Si=Si−yi
最后
以上就是无限香水最近收集整理的关于softmax求导的全部内容，更多相关softmax求导内容请搜索靠谱客的其他文章。
发表评论 取消回复