1. 程式人生 > 其它 >神經網路反向傳播演算法公式推導

神經網路反向傳播演算法公式推導

規定

  • \(y_{ij}\)為第\(i\)層網路第\(j\)個神經元的輸出.
  • \(t_i\)為輸出層第\(i\)個輸出.
  • \(n_i\)為第\(i\)層網路的神經元數量.
  • 啟用函式\(\sigma(x)=Sigmod(x)=\frac{1}{1+e^{-x}}\),因此\(\frac{\partial \sigma(x)}{\partial x}=\sigma(x)[1-\sigma(x)]\).
  • \(E\)代表誤差,即\(E=\sum_{i=1}^{2}(y_{3i}-t_i)^{2}\).
  • \(\nabla_{ijk}\)為第\(i\)層網路第\(j\)個神經元,它對上一層網路的第\(k\)
    個神經元的\(\omega\)的梯度值.

公式推導

現在以\(11\rightarrow 21 \rightarrow 31\)該過程為例,同時匯出通項公式。

1.\(net_{ij}\)

\(net_{21}=\sum_{i=1}^{2}(\omega_{21i}y_{1i})\).

\(net_{31}=\sum_{i=1}^{3}(\omega_{31i}y_{2i})\).

因此通項公式為:\(net_{ij}=\sum_{k=1}^{n_{i-1}}(\omega_{ijk}y_{i-1,k})\), \(y_{ij}=\sigma(net_{ij})\).

2.\(y_{ij}\)

\(y_{21}=\sigma(net_{21})\).

\(y_{31}=\sigma(net_{31})\).

因此通項公式為:\(y_{ij}=\sigma(net_{ij})\).

3.誤差\(E\)

\(E=\sum_{i=1}^{2}(y_{3i}-t_{i})^2\).

4.\(\nabla_{3ij}\)(輸出層各\(\omega\)的梯度值)

\(\frac{\partial E}{\partial net_{31}}=\frac{E}{\partial y_{31}}\cdot \frac{\partial y_{31}}{\partial {net_{31}}}=2(y_{31}-t_1)y_{31}(1-y_{31})\)

.

\(\therefore \frac{E}{\partial net_{3i}}=2(y_{3i}-t_i)y_{3i}(1-y_{3i})\).

\(\nabla_{311}=\frac{\partial E}{\partial net_{31}} \cdot \frac{\partial net_{31}}{\partial \omega_{311}}=\frac{\partial E}{\partial net_{31}}\cdot y_{21}\).

因此通項公式為:\(\nabla_{3ij}=\frac{\partial E}{\partial net_{3i}}\cdot \frac{\partial net_{3i}}{\partial \omega_{3ij}}=2(y_{3i}-t_{i})y_{3i}(1-y_{3i})y_{2i}\).

5.\(\nabla_{2ij}\)(隱藏層各\(\omega\)的梯度值)

\(\nabla_{211}=\frac{\partial E}{\partial \omega_{211}}=\frac{\partial E}{\partial net_{31}}\cdot \frac{\partial net_{31}}{\partial y_{21}}\cdot \frac{\partial y_{21}}{\partial net_{21}}\cdot \frac{\partial{net_{21}}}{\partial \omega_{211}}+\frac{\partial E}{\partial net_{32}}\cdot \frac{\partial net_{32}}{\partial y_{21}}\cdot \frac{\partial y_{21}}{\partial net_{21}}\cdot \frac{\partial{net_{21}}}{\partial \omega_{211}}\\=\sum_{i=1}^{2}(\frac{\partial E}{\partial net_{3i}}\cdot \frac{\partial net_{3i}}{\partial y_{2i}})\cdot \frac{\partial y_{21}}{\partial net_{21}} \cdot \frac{\partial net_{21}}{\partial \omega_{211}}\\=\sum_{i=1}^{2}(\frac{\partial E}{\partial net_{3i}}\cdot \omega_{3i1})\cdot y_{21}(1-y_{21})\omega_{21}\)

因此通項公式為:\(\nabla_{2ij}=\frac{\partial E}{\partial \omega_{2ij}}=\sum_{k=1}^{2}(\frac{\partial E}{\partial net_{3k}}\cdot \omega_{3ki})\cdot y_{2i}(1-y_{2i})\omega_{2ij}\).