深度学习的正、反向传播与梯度下降的实例

Myth丶恋晨 2022-12-23 06:26 146阅读 0赞

采用3层的全连接网络，首先正向传播一次并计算误差，之后反向传播一次更新参数，最后再次正向传播并计算误差

全连接网络的具体结构如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0Jva21hbg_size_16_color_FFFFFF_t_70][]

输入参数(input\_param)：![\\begin\{vmatrix\} 2 & 0 & 3\\\\ 0 & 1 & 0 \\end\{vmatrix\}][begin_vmatrix_ 2 _ 0 _ 3_ 0 _ 1 _ 0 _end_vmatrix]

初始化第一层权重(w1)：![\\begin\{vmatrix\} 2 & 3\\\\ 0 & 1\\\\ 1 & 3 \\end\{vmatrix\}][begin_vmatrix_ 2 _ 3_ 0 _ 1_ 1 _ 3 _end_vmatrix]

初始化第二层参数(w2):![\\begin\{vmatrix\} 1 & 2\\\\ 3 & 4 \\end\{vmatrix\}][begin_vmatrix_ 1 _ 2_ 3 _ 4 _end_vmatrix]

给定预期的输出(output):![\\begin\{vmatrix\} 50 & 71\\\\ 1 & 3 \\end\{vmatrix\}][begin_vmatrix_ 50 _ 71_ 1 _ 3 _end_vmatrix]

## **1. 第一次正向传播及计算误差** ##

进行第一次正向传播得到输出(pred)：

pred = input\_param \* w1 \* w2

![\\begin\{vmatrix\} 2 & 0 & 3\\\\ 0 & 1 & 0 \\end\{vmatrix\} \\times \\begin\{vmatrix\} 2 & 3\\\\ 0 & 1\\\\ 1 & 3 \\end\{vmatrix\} \\times \\begin\{vmatrix\} 1 & 2\\\\ 3 & 4 \\end\{vmatrix\} = \\begin\{vmatrix\} 52 & 74\\\\ 3 & 4 \\end\{vmatrix\}][begin_vmatrix_ 2 _ 0 _ 3_ 0 _ 1 _ 0 _end_vmatrix_ _times _begin_vmatrix_ 2 _ 3_ 0 _ 1_ 1 _ 3 _end_vmatrix_ _times _begin_vmatrix_ 1 _ 2_ 3 _ 4 _end_vmatrix_ _ _begin_vmatrix_ 52 _ 74_ 3 _ 4 _end_vmatrix]

其中隐藏层(hidden)在第一次正向传播中各个参数为：

hidden = input\_param \* w1

![\\begin\{vmatrix\} 2 & 0 & 3\\\\ 0 & 1 & 0 \\end\{vmatrix\} \\times \\begin\{vmatrix\} 2 & 3\\\\ 0 & 1\\\\ 1 & 3 \\end\{vmatrix\} = \\begin\{vmatrix\} 7 & 15\\\\ 0 & 1 \\end\{vmatrix\}][begin_vmatrix_ 2 _ 0 _ 3_ 0 _ 1 _ 0 _end_vmatrix_ _times _begin_vmatrix_ 2 _ 3_ 0 _ 1_ 1 _ 3 _end_vmatrix_ _ _begin_vmatrix_ 7 _ 15_ 0 _ 1 _end_vmatrix]

计算第一次正向传播得出的结果和预期输出之间的误差(grad\_pred)：

grad\_pred = pred - output

![\\begin\{vmatrix\} 54 & 74\\\\ 3 & 4 \\end\{vmatrix\} - \\begin\{vmatrix\} 50 & 71\\\\ 1 & 3 \\end\{vmatrix\} = \\begin\{vmatrix\} 2 & 3\\\\ 2 & 1 \\end\{vmatrix\}][begin_vmatrix_ 54 _ 74_ 3 _ 4 _end_vmatrix_ - _begin_vmatrix_ 50 _ 71_ 1 _ 3 _end_vmatrix_ _ _begin_vmatrix_ 2 _ 3_ 2 _ 1 _end_vmatrix]

**误差的范数为：4.1306**

## **2. 第一次反向传播及更新参数** ##

第二层权重的梯度(grad\_w2)：

grad\_w2 = hidden' \* grad\_pred  (注意hidden需要转置)

![\\begin\{vmatrix\} 7 & 0\\\\ 15 & 1 \\end\{vmatrix\} \\times \\begin\{vmatrix\} 2 & 3\\\\ 2 & 1 \\end\{vmatrix\} = \\begin\{vmatrix\} 14 & 21\\\\ 32 & 46 \\end\{vmatrix\}][begin_vmatrix_ 7 _ 0_ 15 _ 1 _end_vmatrix_ _times _begin_vmatrix_ 2 _ 3_ 2 _ 1 _end_vmatrix_ _ _begin_vmatrix_ 14 _ 21_ 32 _ 46 _end_vmatrix]

hidden层的梯度(grad\_hidden):

grad\_hidden = grad\_pred \* w2'   (注意w2需要转置)

![\\begin\{vmatrix\} 2 & 3\\\\ 2 & 1 \\end\{vmatrix\} \\times \\begin\{vmatrix\} 1 & 3\\\\ 2 & 4 \\end\{vmatrix\} = \\begin\{vmatrix\} 8 & 18\\\\ 4 & 10 \\end\{vmatrix\}][begin_vmatrix_ 2 _ 3_ 2 _ 1 _end_vmatrix_ _times _begin_vmatrix_ 1 _ 3_ 2 _ 4 _end_vmatrix_ _ _begin_vmatrix_ 8 _ 18_ 4 _ 10 _end_vmatrix]

第一层权重的梯度(grad\_w1):

grad\_w1 = input\_param' \* grad\_hidden   (注意input\_param需要转置)

![\\begin\{vmatrix\} 2 & 0\\\\ 0 & 1\\\\ 3 & 0 \\end\{vmatrix\} \\times \\begin\{vmatrix\} 8 & 18\\\\ 4 & 10 \\end\{vmatrix\} = \\begin\{vmatrix\} 16 & 36\\\\ 4 & 10\\\\ 24 & 54 \\end\{vmatrix\}][begin_vmatrix_ 2 _ 0_ 0 _ 1_ 3 _ 0 _end_vmatrix_ _times _begin_vmatrix_ 8 _ 18_ 4 _ 10 _end_vmatrix_ _ _begin_vmatrix_ 16 _ 36_ 4 _ 10_ 24 _ 54 _end_vmatrix]

更新神经网络的两层权重(w1和w2):

new\_w1 = w1 - learning\_rate \* grad\_w1

![\\begin\{vmatrix\} 2 & 3\\\\ 0 & 1\\\\ 1 & 3 \\end\{vmatrix\} - 0.002 \\times \\begin\{vmatrix\} 16 & 36\\\\ 4 & 10\\\\ 24 & 54 \\end\{vmatrix\}=\\begin\{vmatrix\} 1.968 & 2.928\\\\ -0.008 & 0.98\\\\ 0.952 & 2.892 \\end\{vmatrix\}][begin_vmatrix_ 2 _ 3_ 0 _ 1_ 1 _ 3 _end_vmatrix_ - 0.002 _times _begin_vmatrix_ 16 _ 36_ 4 _ 10_ 24 _ 54 _end_vmatrix_begin_vmatrix_ 1.968 _ 2.928_ -0.008 _ 0.98_ 0.952 _ 2.892 _end_vmatrix]

new\_w2 = w2 - learning\_rate \* grad\_w2

![\\begin\{vmatrix\} 1 & 2\\\\ 3 & 4 \\end\{vmatrix\} - 0.002 \\times \\begin\{vmatrix\} 14 & 21\\\\ 32 & 46 \\end\{vmatrix\} = \\begin\{vmatrix\} 0.972 & 1.958\\\\ 2.936 & 3.908 \\end\{vmatrix\}][begin_vmatrix_ 1 _ 2_ 3 _ 4 _end_vmatrix_ - 0.002 _times _begin_vmatrix_ 14 _ 21_ 32 _ 46 _end_vmatrix_ _ _begin_vmatrix_ 0.972 _ 1.958_ 2.936 _ 3.908 _end_vmatrix]

## 3. **第一次正向传播及计算误差** ##

进行第二次正向传播得到输出(new\_pred)：

![\\begin\{vmatrix\} 2 & 0 & 3\\\\ 0 & 1 & 0 \\end\{vmatrix\} \\times \\begin\{vmatrix\} 1.968 & 2.928\\\\ -0.008 & 0.98\\\\ 0.952 & 2.892 \\end\{vmatrix\} \\times \\begin\{vmatrix\} 0.972 & 1.958\\\\ 2.936 & 3.908 \\end\{vmatrix\} = \\begin\{vmatrix\} 49.2678 & 70.0898\\\\ 2.8695 & 3.8142 \\end\{vmatrix\}][begin_vmatrix_ 2 _ 0 _ 3_ 0 _ 1 _ 0 _end_vmatrix_ _times _begin_vmatrix_ 1.968 _ 2.928_ -0.008 _ 0.98_ 0.952 _ 2.892 _end_vmatrix_ _times _begin_vmatrix_ 0.972 _ 1.958_ 2.936 _ 3.908 _end_vmatrix_ _ _begin_vmatrix_ 49.2678 _ 70.0898_ 2.8695 _ 3.8142 _end_vmatrix]

计算第二次正向传播得出的结果和预期输出之间的误差(new\_grad\_pred)：

![\\begin\{vmatrix\} 49.2678 & 70.0898\\\\ 2.8695 & 3.8142 \\end\{vmatrix\} - \\begin\{vmatrix\} 50 & 71\\\\ 1 & 3 \\end\{vmatrix\} = \\begin\{vmatrix\} -0.7322 & -0.9102\\\\ 1.8695 & 0.8142 \\end\{vmatrix\}][begin_vmatrix_ 49.2678 _ 70.0898_ 2.8695 _ 3.8142 _end_vmatrix_ - _begin_vmatrix_ 50 _ 71_ 1 _ 3 _end_vmatrix_ _ _begin_vmatrix_ -0.7322 _ -0.9102_ 1.8695 _ 0.8142 _end_vmatrix]

**此实误差的范数为：2.3004**

### **因此，通过一次反向传播更新参数之后，神经网络输出的结果和预期解之间的误差减小了(4.1306 -> 2.3004)** ###

**注：在实际仿真的时候，学习率采用0.05，0.005都会导致系统无法收敛，误差越来越大。当使用0.003作为学习率时，系统误差下降。**

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0Jva21hbg_size_16_color_FFFFFF_t_70]: /images/20221120/7a438ae8553c4458802d98c3e81d267d.png
[begin_vmatrix_ 2 _ 0 _ 3_ 0 _ 1 _ 0 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%202%20%26%200%20%26%203%5C%5C%200%20%26%201%20%26%200%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 2 _ 3_ 0 _ 1_ 1 _ 3 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%202%20%26%203%5C%5C%200%20%26%201%5C%5C%201%20%26%203%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 1 _ 2_ 3 _ 4 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%201%20%26%202%5C%5C%203%20%26%204%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 50 _ 71_ 1 _ 3 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%2050%20%26%2071%5C%5C%201%20%26%203%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 2 _ 0 _ 3_ 0 _ 1 _ 0 _end_vmatrix_ _times _begin_vmatrix_ 2 _ 3_ 0 _ 1_ 1 _ 3 _end_vmatrix_ _times _begin_vmatrix_ 1 _ 2_ 3 _ 4 _end_vmatrix_ _ _begin_vmatrix_ 52 _ 74_ 3 _ 4 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%202%20%26%200%20%26%203%5C%5C%200%20%26%201%20%26%200%20%5Cend%7Bvmatrix%7D%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%202%20%26%203%5C%5C%200%20%26%201%5C%5C%201%20%26%203%20%5Cend%7Bvmatrix%7D%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%201%20%26%202%5C%5C%203%20%26%204%20%5Cend%7Bvmatrix%7D%20%3D%20%5Cbegin%7Bvmatrix%7D%2052%20%26%2074%5C%5C%203%20%26%204%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 2 _ 0 _ 3_ 0 _ 1 _ 0 _end_vmatrix_ _times _begin_vmatrix_ 2 _ 3_ 0 _ 1_ 1 _ 3 _end_vmatrix_ _ _begin_vmatrix_ 7 _ 15_ 0 _ 1 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%202%20%26%200%20%26%203%5C%5C%200%20%26%201%20%26%200%20%5Cend%7Bvmatrix%7D%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%202%20%26%203%5C%5C%200%20%26%201%5C%5C%201%20%26%203%20%5Cend%7Bvmatrix%7D%20%3D%20%5Cbegin%7Bvmatrix%7D%207%20%26%2015%5C%5C%200%20%26%201%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 54 _ 74_ 3 _ 4 _end_vmatrix_ - _begin_vmatrix_ 50 _ 71_ 1 _ 3 _end_vmatrix_ _ _begin_vmatrix_ 2 _ 3_ 2 _ 1 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%2054%20%26%2074%5C%5C%203%20%26%204%20%5Cend%7Bvmatrix%7D%20-%20%5Cbegin%7Bvmatrix%7D%2050%20%26%2071%5C%5C%201%20%26%203%20%5Cend%7Bvmatrix%7D%20%3D%20%5Cbegin%7Bvmatrix%7D%202%20%26%203%5C%5C%202%20%26%201%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 7 _ 0_ 15 _ 1 _end_vmatrix_ _times _begin_vmatrix_ 2 _ 3_ 2 _ 1 _end_vmatrix_ _ _begin_vmatrix_ 14 _ 21_ 32 _ 46 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%207%20%26%200%5C%5C%2015%20%26%201%20%5Cend%7Bvmatrix%7D%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%202%20%26%203%5C%5C%202%20%26%201%20%5Cend%7Bvmatrix%7D%20%3D%20%5Cbegin%7Bvmatrix%7D%2014%20%26%2021%5C%5C%2032%20%26%2046%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 2 _ 3_ 2 _ 1 _end_vmatrix_ _times _begin_vmatrix_ 1 _ 3_ 2 _ 4 _end_vmatrix_ _ _begin_vmatrix_ 8 _ 18_ 4 _ 10 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%202%20%26%203%5C%5C%202%20%26%201%20%5Cend%7Bvmatrix%7D%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%201%20%26%203%5C%5C%202%20%26%204%20%5Cend%7Bvmatrix%7D%20%3D%20%5Cbegin%7Bvmatrix%7D%208%20%26%2018%5C%5C%204%20%26%2010%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 2 _ 0_ 0 _ 1_ 3 _ 0 _end_vmatrix_ _times _begin_vmatrix_ 8 _ 18_ 4 _ 10 _end_vmatrix_ _ _begin_vmatrix_ 16 _ 36_ 4 _ 10_ 24 _ 54 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%202%20%26%200%5C%5C%200%20%26%201%5C%5C%203%20%26%200%20%5Cend%7Bvmatrix%7D%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%208%20%26%2018%5C%5C%204%20%26%2010%20%5Cend%7Bvmatrix%7D%20%3D%20%5Cbegin%7Bvmatrix%7D%2016%20%26%2036%5C%5C%204%20%26%2010%5C%5C%2024%20%26%2054%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 2 _ 3_ 0 _ 1_ 1 _ 3 _end_vmatrix_ - 0.002 _times _begin_vmatrix_ 16 _ 36_ 4 _ 10_ 24 _ 54 _end_vmatrix_begin_vmatrix_ 1.968 _ 2.928_ -0.008 _ 0.98_ 0.952 _ 2.892 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%202%20%26%203%5C%5C%200%20%26%201%5C%5C%201%20%26%203%20%5Cend%7Bvmatrix%7D%20-%200.002%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%2016%20%26%2036%5C%5C%204%20%26%2010%5C%5C%2024%20%26%2054%20%5Cend%7Bvmatrix%7D%3D%5Cbegin%7Bvmatrix%7D%201.968%20%26%202.928%5C%5C%20-0.008%20%26%200.98%5C%5C%200.952%20%26%202.892%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 1 _ 2_ 3 _ 4 _end_vmatrix_ - 0.002 _times _begin_vmatrix_ 14 _ 21_ 32 _ 46 _end_vmatrix_ _ _begin_vmatrix_ 0.972 _ 1.958_ 2.936 _ 3.908 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%201%20%26%202%5C%5C%203%20%26%204%20%5Cend%7Bvmatrix%7D%20-%200.002%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%2014%20%26%2021%5C%5C%2032%20%26%2046%20%5Cend%7Bvmatrix%7D%20%3D%20%5Cbegin%7Bvmatrix%7D%200.972%20%26%201.958%5C%5C%202.936%20%26%203.908%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 2 _ 0 _ 3_ 0 _ 1 _ 0 _end_vmatrix_ _times _begin_vmatrix_ 1.968 _ 2.928_ -0.008 _ 0.98_ 0.952 _ 2.892 _end_vmatrix_ _times _begin_vmatrix_ 0.972 _ 1.958_ 2.936 _ 3.908 _end_vmatrix_ _ _begin_vmatrix_ 49.2678 _ 70.0898_ 2.8695 _ 3.8142 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%202%20%26%200%20%26%203%5C%5C%200%20%26%201%20%26%200%20%5Cend%7Bvmatrix%7D%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%201.968%20%26%202.928%5C%5C%20-0.008%20%26%200.98%5C%5C%200.952%20%26%202.892%20%5Cend%7Bvmatrix%7D%20%5Ctimes%20%5Cbegin%7Bvmatrix%7D%200.972%20%26%201.958%5C%5C%202.936%20%26%203.908%20%5Cend%7Bvmatrix%7D%20%3D%20%5Cbegin%7Bvmatrix%7D%2049.2678%20%26%2070.0898%5C%5C%202.8695%20%26%203.8142%20%5Cend%7Bvmatrix%7D
[begin_vmatrix_ 49.2678 _ 70.0898_ 2.8695 _ 3.8142 _end_vmatrix_ - _begin_vmatrix_ 50 _ 71_ 1 _ 3 _end_vmatrix_ _ _begin_vmatrix_ -0.7322 _ -0.9102_ 1.8695 _ 0.8142 _end_vmatrix]: https://private.codecogs.com/gif.latex?%5Cbegin%7Bvmatrix%7D%2049.2678%20%26%2070.0898%5C%5C%202.8695%20%26%203.8142%20%5Cend%7Bvmatrix%7D%20-%20%5Cbegin%7Bvmatrix%7D%2050%20%26%2071%5C%5C%201%20%26%203%20%5Cend%7Bvmatrix%7D%20%3D%20%5Cbegin%7Bvmatrix%7D%20-0.7322%20%26%20-0.9102%5C%5C%201.8695%20%26%200.8142%20%5Cend%7Bvmatrix%7D