吴恩达机器学习：神经网络 | 反向传播算法

逃离我推掉我的手 2022-01-20 02:05 323阅读 0赞

上一周我们学习了 [**神经网络 | 多分类问题**][Link 1]。我们分别使用 **逻辑回归** 和 **神经网络** 来解决多分类问题，并了解到在特征数非常多的情况下，神经网络是更为有效的方法。这周的课程会给出训练 **神经网络** 所使用的 **代价函数**，并使用 **反向传播** 算法来计算梯度。笔者会给出 **反向传播** 算法中重要的思路和推导，但不会包含所有的计算步骤。

点击 [**课程视频**][Link 2] 你就能不间断地学习 Ng 的课程，关于课程作业的 Python 代码已经放到了 Github 上，点击 [**课程代码**][Link 3] 就能去 Github 查看（ 无法访问 Github 的话可以点击 [**Coding**][Coding] 查看 ），代码中的错误和改进欢迎大家指出。

以下是 Ng 机器学习课程第四周的笔记。

### 代价函数 ###

假设我们的多分类问题有 ![K][] 个分类，神经网络共有 ![L][] 层，每一层的神经元个数为 ![s\_l][s_l]，那么神经网络的 **代价函数** 为：

![J(\\Theta)= -\\frac\{1\}\{m\}\\sum\\limits\_\{i=1\}^\{m\}\\sum\\limits\_\{k=1\}^\{K\}\\Big(y\_k^\{(i)\}log\\big(h\_\\Theta(x^\{(i)\})\\big)\_k+(1-y\_k^\{(i)\})log\\big(1-\\big(h\_\\Theta(x^\{(i)\})\\big)\_k\\big)\\Big) \\\\ + \\frac\{\\lambda\}\{2m\}\\sum\\limits\_\{l=1\}^\{L-1\}\\sum\\limits\_\{i=1\}^\{s\_l\}\\sum\\limits\_\{j=1\}^\{s\_\{l+1\}\}\\big(\\Theta\_\{ji\}^\{(l)\}\\big)^2][J_Theta_ -_frac_1_m_sum_limits_i_1_m_sum_limits_k_1_K_Big_y_k_i_log_big_h_Theta_x_i_big_k_1-y_k_i_log_big_1-_big_h_Theta_x_i_big_k_big_Big_ _ _ _frac_lambda_2m_sum_limits_l_1_L-1_sum_limits_i_1_s_l_sum_limits_j_1_s_l_1_big_Theta_ji_l_big_2]

其中的第二项为 **正则化** 项，是网络中所有权值的平方和。第一项与逻辑回归中的 **代价函数** 类似，但这里我们需要累加所有输出神经元的误差。

### 梯度计算 ###

为了能够使用 **梯度下降** 算法来训练网络，我们需要计算代价函数的梯度。一种很直观的方法就是使用数值计算，对于某个 ![\\Theta\_\{ij\}][Theta_ij]，给它加上减去一个很小的量 ![\\epsilon][epsilon] 来计算梯度：

![\\frac\{\\partial J(\\theta)\}\{\\partial\\theta\_j\} \\approx \\frac\{J(\\theta\_1,\\cdots,\\theta\_j+\\epsilon,\\cdots,\\theta\_n)-J(\\theta\_1,\\cdots,\\theta\_j-\\epsilon,\\cdots,\\theta\_n)\}\{2\\epsilon\}][frac_partial J_theta_partial_theta_j_ _approx _frac_J_theta_1_cdots_theta_j_epsilon_cdots_theta_n_-J_theta_1_cdots_theta_j-_epsilon_cdots_theta_n_2_epsilon]

但稍微分析一下算法的复杂度就能知道，这样的方法十分缓慢。对于每一组数据，我们需要计算所有权值的梯度，\*\*总的计算次数 = 训练数据个数 x 网络权值个数 x 前向传播计算次数 \*\*。在通常情况下这样的复杂度是无法接受的，所以我们仅使用这个方法来验证 **反向传播** 算法计算的梯度是否正确。

### 链式法则 ###

为了能够理解之后对于 **反向传播** 公式的推导，我们首先要了解一个关于多元复合函数求导的 **链式法则**。对于多元函数 ![z=f(u,v)][z_f_u_v]，其中 ![u=h(x,y)][u_h_x_y]，![v=g(x,y)][v_g_x_y]，那么：

![\\frac\{\\partial z\}\{\\partial x\} = \\frac\{\\partial z\}\{\\partial u\}\\frac\{\\partial u\}\{\\partial x\} + \\frac\{\\partial z\}\{\\partial v\}\\frac\{\\partial v\}\{\\partial x\} \\\\ \\frac\{\\partial z\}\{\\partial y\} = \\frac\{\\partial z\}\{\\partial u\}\\frac\{\\partial u\}\{\\partial y\} + \\frac\{\\partial z\}\{\\partial v\}\\frac\{\\partial v\}\{\\partial y\}][frac_partial z_partial x_ _ _frac_partial z_partial u_frac_partial u_partial x_ _ _frac_partial z_partial v_frac_partial v_partial x_ _ _frac_partial z_partial y_ _ _frac_partial z_partial u_frac_partial u_partial y_ _ _frac_partial z_partial v_frac_partial v_partial y]

**链式法则** 告诉我们有多个层次的多元复合函数，下一层次的导数可以由上一层次推得。

上图中笔者有意多加了一层，这里 ![p][] 是 ![u,v][u_v] 的函数， ![q][] 是 ![u,v][u_v] 的函数， ![z][] 是 ![p,q][p_q] 的函数。对于要计算的 ![\\frac\{\\partial z\}\{\\partial x\}][frac_partial z_partial x] 与 ![\\frac\{\\partial z\}\{\\partial y\}][frac_partial z_partial y]，上式仍成立，原因是我们可以把 ![z][] 看作 ![u,v][u_v] 的函数。这相当于我们把：

![\\frac\{\\partial z\}\{\\partial x\} = \\frac\{\\partial z\}\{\\partial p\} \\frac\{\\partial p\}\{\\partial u\} \\frac\{\\partial u\}\{\\partial x\} + \\frac\{\\partial z\}\{\\partial p\} \\frac\{\\partial p\}\{\\partial v\} \\frac\{\\partial v\}\{\\partial x\} + \\frac\{\\partial z\}\{\\partial q\} \\frac\{\\partial q\}\{\\partial u\} \\frac\{\\partial u\}\{\\partial x\} + \\frac\{\\partial z\}\{\\partial q\} \\frac\{\\partial q\}\{\\partial v\} \\frac\{\\partial v\}\{\\partial x\}][frac_partial z_partial x_ _ _frac_partial z_partial p_ _frac_partial p_partial u_ _frac_partial u_partial x_ _ _frac_partial z_partial p_ _frac_partial p_partial v_ _frac_partial v_partial x_ _ _frac_partial z_partial q_ _frac_partial q_partial u_ _frac_partial u_partial x_ _ _frac_partial z_partial q_ _frac_partial q_partial v_ _frac_partial v_partial x]

简化为了只与上一层相关，利用上一层计算完成的结果 ![\\frac\{\\partial z\}\{\\partial u\}][frac_partial z_partial u] 和 ![\\frac\{\\partial z\}\{\\partial v\}][frac_partial z_partial v] 而不用从头算起：

![\\frac\{\\partial z\}\{\\partial x\} = \\frac\{\\partial z\}\{\\partial u\}\\frac\{\\partial u\}\{\\partial x\} + \\frac\{\\partial z\}\{\\partial v\}\\frac\{\\partial v\}\{\\partial x\}][frac_partial z_partial x_ _ _frac_partial z_partial u_frac_partial u_partial x_ _ _frac_partial z_partial v_frac_partial v_partial x]

一般的，对于函数 ![y][]，如果它能看做 ![z\_1,z\_2 \\cdots,z\_n][z_1_z_2 _cdots_z_n] 的函数，而 ![z\_i][z_i] 为 ![t][] 的函数，则：

![\\frac\{\\partial y\}\{\\partial t\} = \\sum\\limits\_\{i=1\}^\{n\} \\frac\{\\partial y\}\{\\partial z\_i\} \\frac\{\\partial z\_i\}\{\\partial t\}][frac_partial y_partial t_ _ _sum_limits_i_1_n_ _frac_partial y_partial z_i_ _frac_partial z_i_partial t]

神经网络就是一个层次很多的多元函数，我们可以隐约从 **链式法则** 中感觉到反向传播的意味。

### 公式推导 ###

为了施展 **反向传播** 的魔法，我们首要要引入一个中间变量 ![\\delta][delta]，定义为：

![\\delta\_j^\{l\} = \\frac\{\\partial J\}\{\\partial z\_j^\{l\}\}][delta_j_l_ _ _frac_partial J_partial z_j_l]

其中 ![l][] 为第几层，![j][] 表示第 ![l][] 层的第几个神经元，![z][] 为上次课程提到的中间变量（ 为了让式子看上去更清晰，**反向传播** 中的公式上标不使用括号 ）。![\\delta][delta] 被称为第 ![l][] 层第 ![j][] 个神经元的误差。**反向传播** 就是先计算每一层每个神经元的误差，然后通过误差来得到梯度的。

首先来看输出层的误差：

![\\delta^L\_j = \\frac\{\\partial J\}\{\\partial z^L\_j\}][delta_L_j _ _frac_partial J_partial z_L_j]

对它使用 **链式法则** 得到：

![\\delta^L\_j = \\sum\\limits\_\{k=1\}^\{K\} \\frac\{\\partial J\}\{\\partial a^L\_k\} \\frac\{\\partial a^L\_k\}\{\\partial z^L\_j\}][delta_L_j _ _sum_limits_k_1_K_ _frac_partial J_partial a_L_k_ _frac_partial a_L_k_partial z_L_j]

而只有当 ![k==j][k_j] 时，右边部分才不为 ![0][]，所以：

![\\delta^L\_j = \\frac\{\\partial J\}\{\\partial a^L\_j\} \\frac\{\\partial a^L\_j\}\{\\partial z^L\_j\} = \\frac\{\\partial J\}\{\\partial a^L\_j\} g'(z\_j^L)=a\_j^L-y\_j^L\\tag\{1\}][delta_L_j _ _frac_partial J_partial a_L_j_ _frac_partial a_L_j_partial z_L_j_ _ _frac_partial J_partial a_L_j_ g_z_j_L_a_j_L-y_j_L_tag_1]

对于其它层的误差：

![\\delta^l\_j = \\frac\{\\partial J\}\{\\partial z^l\_j\}][delta_l_j _ _frac_partial J_partial z_l_j]

使用 **链式法则**：

![\\delta^l\_j = \\sum\\limits\_\{k=1\}^\{s\_\{l+1\}\} \\frac\{\\partial J\}\{\\partial z\_k^\{l+1\}\} \\frac\{\\partial z\_k^\{l+1\}\}\{\\partial z^l\_j\} = \\sum\\limits\_\{k=1\}^\{s\_\{l+1\}\} \\delta^\{l+1\}\_k \\frac\{\\partial z\_k^\{l+1\}\}\{\\partial z^l\_j\}][delta_l_j _ _sum_limits_k_1_s_l_1_ _frac_partial J_partial z_k_l_1_ _frac_partial z_k_l_1_partial z_l_j_ _ _sum_limits_k_1_s_l_1_ _delta_l_1_k _frac_partial z_k_l_1_partial z_l_j]

而其中：

![z\_k^\{l+1\} = \\sum\\limits\_\{p=1\}^\{s\_l\} \\Theta\_\{kp\}^\{l\} g(z\_p^l)+b\_k^\{l\}][z_k_l_1_ _ _sum_limits_p_1_s_l_ _Theta_kp_l_ g_z_p_l_b_k_l]

求偏导得：

![\\frac\{\\partial z^\{l+1\}\_k\}\{\\partial z^l\_j\} = \\Theta\_\{kj\}^\{l\} g'(z^l\_j)][frac_partial z_l_1_k_partial z_l_j_ _ _Theta_kj_l_ g_z_l_j]

所以：

![\\delta^l\_j = \\sum\\limits\_\{k=1\}^\{s\_\{l+1\}\} \\Theta\_\{kj\}^\{l\} \\delta^\{l+1\}\_k g'(z^l\_j) = \\sum\\limits\_\{k=1\}^\{s\_\{l+1\}\} \\Theta\_\{kj\}^\{l\} \\delta^\{l+1\}\_k a\_j^l (1-a\_j^l) \\tag\{2\}][delta_l_j _ _sum_limits_k_1_s_l_1_ _Theta_kj_l_ _delta_l_1_k g_z_l_j_ _ _sum_limits_k_1_s_l_1_ _Theta_kj_l_ _delta_l_1_k a_j_l _1-a_j_l_ _tag_2]

最后同样使用 **链式法则** 来计算：

![\\frac\{\\partial J\}\{\\Theta\_\{ij\}^\{l\}\} = \\sum\\limits\_\{k=1\}^\{s\_\{l+1\}\} \\frac\{\\partial J\}\{\\partial z\_k^\{l+1\}\} \\frac\{\\partial z\_k^\{l+1\}\}\{\\partial \\Theta\_\{ij\}^\{l\}\} = \\sum\\limits\_\{k=1\}^\{s\_\{l\}\} \\delta^\{l+1\}\_k \\frac\{\\partial z\_k^\{l+1\}\}\{\\partial \\Theta\_\{ij\}^\{l\}\}][frac_partial J_Theta_ij_l_ _ _sum_limits_k_1_s_l_1_ _frac_partial J_partial z_k_l_1_ _frac_partial z_k_l_1_partial _Theta_ij_l_ _ _sum_limits_k_1_s_l_ _delta_l_1_k _frac_partial z_k_l_1_partial _Theta_ij_l]

由于：

![z\_k^\{l+1\} = \\sum\\limits\_\{p=1\}^\{s\_l\} \\Theta\_\{kp\}^\{l\} g(z\_p^l)+b\_k^\{l\}][z_k_l_1_ _ _sum_limits_p_1_s_l_ _Theta_kp_l_ g_z_p_l_b_k_l]

只有当 ![k=i][k_i]，![p=j][p_j] 时留下一项：

![\\frac\{\\partial J\}\{\\Theta\_\{ij\}^\{l\}\} = g(z\_j^l) \\delta^\{l+1\}\_i = a\_j^l \\delta^\{l+1\}\_i \\tag\{3\}][frac_partial J_Theta_ij_l_ _ g_z_j_l_ _delta_l_1_i _ a_j_l _delta_l_1_i _tag_3]

### 反向传播 ###

有了 (1) (2) (3) 式，就可以来完成 **反向传播** 算法了（ 需要注意的是刚才所推导的式子都是针对一组训练数据而言的 ）。

> 1.  对于所有的 ![l, i, j][l_ i_ j] 初始化 ![\\Delta\_\{ij\}^l = 0][Delta_ij_l _ 0]
> 2.  对于 ![m][] 组训练数据，![k][] 从 ![1][] 取到 ![m][]：
> 
>  *  令 ![a^1 = x^\{(k)\}][a_1 _ x_k]
>  *  前向传播，计算各层激活向量 ![a^\{l\}][a_l]
>  *  使用 (1) 式，计算输出层误差 ![\\delta^L][delta_L]
>  *  使用 (2) 式，计算其它层误差 ![\\delta^\{L-1\},\\delta^\{L-2\},...,\\delta^\{2\}][delta_L-1_delta_L-2_..._delta_2]
>  *  使用 (3) 式，累加 ![\\Delta\_\{ij\}^l][Delta_ij_l]，![\\Delta\_\{ij\}^l:=\\Delta\_\{ij\}^l+a\_j^l\\delta\_i^\{l+1\}][Delta_ij_l_Delta_ij_l_a_j_l_delta_i_l_1]
> 
> 1.  计算梯度矩阵：
> 
> ![D^\{l\}\_\{ij\} = \\begin\{cases\} \\dfrac\{1\}\{m\}\\Delta^\{l\}\_\{ij\} + \\dfrac\{\\lambda\}\{m\}\\Theta^\{l\}\_\{ij\} & \\mbox\{if $j \\neq 0$\} \\\\ \\dfrac\{1\}\{m\}\\Delta\_\{ij\}^\{l\} & \\mbox\{if $j=0$\} \\end\{cases\}][D_l_ij_ _ _begin_cases_ _dfrac_1_m_Delta_l_ij_ _ _dfrac_lambda_m_Theta_l_ij_ _ _mbox_if _j _neq 0_ _ _dfrac_1_m_Delta_ij_l_ _ _mbox_if _j_0_ _end_cases]
> 
> 1.  更新权值 ![\\Theta^l := \\Theta^l+ \\alpha D^l][Theta_l _ _Theta_l_ _alpha D_l]

### 权值初始化 ###

最后提一下权值的初始化。对于神经网络，不能像之前那样使用相同的 0 值来初始化，这会导致每层的 **逻辑单元** 都相同。因此我们使用随机化的初始化方法，使得 ![\-\\delta \\leq \\Theta\_\{ij\}^\{l\} \\leq \\delta][-_delta _leq _Theta_ij_l_ _leq _delta]。

So~，这周学完了 **神经网络** 和它的学习算法，大家有没有觉得很神奇呢？

[Link 1]: https://link.juejin.im?target=http%3A%2F%2Fhertzcat.com%2F2018%2F04%2F07%2Fcoursera-ml-andrewng-nn-multi-class%2F
[Link 2]: https://link.juejin.im?target=https%3A%2F%2Fwww.bilibili.com%2Fvideo%2Fav9912938%3Ffrom%3Dsearch%26amp%3Bseid%3D5092317489110083757
[Link 3]: https://link.juejin.im?target=https%3A%2F%2Fgithub.com%2Fhertzcat%2FCoursera-Machine-Learning
[Coding]: https://link.juejin.im?target=https%3A%2F%2Fcoding.net%2Fu%2Fhertzcat%2Fp%2FCoursera-Machine-Learning%2Fgit%3Fpublic%3Dtrue
[K]: https://juejin.im/equation?tex=K
[L]: https://juejin.im/equation?tex=L
[s_l]: https://juejin.im/equation?tex=s_l
[J_Theta_ -_frac_1_m_sum_limits_i_1_m_sum_limits_k_1_K_Big_y_k_i_log_big_h_Theta_x_i_big_k_1-y_k_i_log_big_1-_big_h_Theta_x_i_big_k_big_Big_ _ _ _frac_lambda_2m_sum_limits_l_1_L-1_sum_limits_i_1_s_l_sum_limits_j_1_s_l_1_big_Theta_ji_l_big_2]: /images/20220120/572daace37bd41d29ca8e650cf2aab36.png
[Theta_ij]: https://juejin.im/equation?tex=%5CTheta_%7Bij%7D
[epsilon]: /images/20220120/25bc2f45235841d0a2cb35ec7b763fa0.png
[frac_partial J_theta_partial_theta_j_ _approx _frac_J_theta_1_cdots_theta_j_epsilon_cdots_theta_n_-J_theta_1_cdots_theta_j-_epsilon_cdots_theta_n_2_epsilon]: /images/20220120/aad6fabe24af4501a817c3c59c161f43.png
[z_f_u_v]: https://juejin.im/equation?tex=z%3Df%28u%2Cv%29
[u_h_x_y]: https://juejin.im/equation?tex=u%3Dh%28x%2Cy%29
[v_g_x_y]: /images/20220120/8963c52574384356bc81005287bf23cd.png
[frac_partial z_partial x_ _ _frac_partial z_partial u_frac_partial u_partial x_ _ _frac_partial z_partial v_frac_partial v_partial x_ _ _frac_partial z_partial y_ _ _frac_partial z_partial u_frac_partial u_partial y_ _ _frac_partial z_partial v_frac_partial v_partial y]: /images/20220120/60a988a8e40b49ee86c1d30152bd2753.png
[p]: https://juejin.im/equation?tex=p
[u_v]: https://juejin.im/equation?tex=u%2Cv
[q]: https://juejin.im/equation?tex=q
[z]: https://juejin.im/equation?tex=z
[p_q]: https://juejin.im/equation?tex=p%2Cq
[frac_partial z_partial x]: https://juejin.im/equation?tex=%5Cfrac%7B%5Cpartial%20z%7D%7B%5Cpartial%20x%7D
[frac_partial z_partial y]: https://juejin.im/equation?tex=%5Cfrac%7B%5Cpartial%20z%7D%7B%5Cpartial%20y%7D
[frac_partial z_partial x_ _ _frac_partial z_partial p_ _frac_partial p_partial u_ _frac_partial u_partial x_ _ _frac_partial z_partial p_ _frac_partial p_partial v_ _frac_partial v_partial x_ _ _frac_partial z_partial q_ _frac_partial q_partial u_ _frac_partial u_partial x_ _ _frac_partial z_partial q_ _frac_partial q_partial v_ _frac_partial v_partial x]: /images/20220120/a7ed47bff9bc44f3903bcf6d371c505f.png
[frac_partial z_partial u]: https://juejin.im/equation?tex=%5Cfrac%7B%5Cpartial%20z%7D%7B%5Cpartial%20u%7D
[frac_partial z_partial v]: /images/20220120/abe5f0d9ca8645cbb6ac3ea69114e45f.png
[frac_partial z_partial x_ _ _frac_partial z_partial u_frac_partial u_partial x_ _ _frac_partial z_partial v_frac_partial v_partial x]: /images/20220120/ba53641ab59340ab8029abdd6c920ec2.png
[y]: https://juejin.im/equation?tex=y
[z_1_z_2 _cdots_z_n]: https://juejin.im/equation?tex=z_1%2Cz_2%20%5Ccdots%2Cz_n
[z_i]: https://juejin.im/equation?tex=z_i
[t]: /images/20220120/038b60d2a3ad4e6e96a33f68d3a7148e.png
[frac_partial y_partial t_ _ _sum_limits_i_1_n_ _frac_partial y_partial z_i_ _frac_partial z_i_partial t]: /images/20220120/853c8de0d96347799ee235294c5ea0fa.png
[delta]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png
[delta_j_l_ _ _frac_partial J_partial z_j_l]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png_j%5E%7Bl%7D%20%3D%20%5Cfrac%7B%5Cpartial%20J%7D%7B%5Cpartial%20z_j%5E%7Bl%7D%7D
[l]: https://juejin.im/equation?tex=l
[j]: /images/20220120/3be6c539585b47b0a4df6761dabe175b.png
[delta_L_j _ _frac_partial J_partial z_L_j]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png%5EL_j%20%3D%20%5Cfrac%7B%5Cpartial%20J%7D%7B%5Cpartial%20z%5EL_j%7D
[delta_L_j _ _sum_limits_k_1_K_ _frac_partial J_partial a_L_k_ _frac_partial a_L_k_partial z_L_j]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png%5EL_j%20%3D%20%5Csum%5Climits_%7Bk%3D1%7D%5E%7BK%7D%20%5Cfrac%7B%5Cpartial%20J%7D%7B%5Cpartial%20a%5EL_k%7D%20%5Cfrac%7B%5Cpartial%20a%5EL_k%7D%7B%5Cpartial%20z%5EL_j%7D
[k_j]: https://juejin.im/equation?tex=k%3D%3Dj
[0]: /images/20220120/016f435d0ae849a1a5bcfaeece55b5ea.png
[delta_L_j _ _frac_partial J_partial a_L_j_ _frac_partial a_L_j_partial z_L_j_ _ _frac_partial J_partial a_L_j_ g_z_j_L_a_j_L-y_j_L_tag_1]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png%5EL_j%20%3D%20%5Cfrac%7B%5Cpartial%20J%7D%7B%5Cpartial%20a%5EL_j%7D%20%5Cfrac%7B%5Cpartial%20a%5EL_j%7D%7B%5Cpartial%20z%5EL_j%7D%20%3D%20%5Cfrac%7B%5Cpartial%20J%7D%7B%5Cpartial%20a%5EL_j%7D%20g'%28z_j%5EL%29%3Da_j%5EL-y_j%5EL%5Ctag%7B1%7D
[delta_l_j _ _frac_partial J_partial z_l_j]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png%5El_j%20%3D%20%5Cfrac%7B%5Cpartial%20J%7D%7B%5Cpartial%20z%5El_j%7D
[delta_l_j _ _sum_limits_k_1_s_l_1_ _frac_partial J_partial z_k_l_1_ _frac_partial z_k_l_1_partial z_l_j_ _ _sum_limits_k_1_s_l_1_ _delta_l_1_k _frac_partial z_k_l_1_partial z_l_j]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png%5El_j%20%3D%20%5Csum%5Climits_%7Bk%3D1%7D%5E%7Bs_%7Bl%2B1%7D%7D%20%5Cfrac%7B%5Cpartial%20J%7D%7B%5Cpartial%20z_k%5E%7Bl%2B1%7D%7D%20%5Cfrac%7B%5Cpartial%20z_k%5E%7Bl%2B1%7D%7D%7B%5Cpartial%20z%5El_j%7D%20%3D%20%5Csum%5Climits_%7Bk%3D1%7D%5E%7Bs_%7Bl%2B1%7D%7D%20%5Cdelta%5E%7Bl%2B1%7D_k%20%5Cfrac%7B%5Cpartial%20z_k%5E%7Bl%2B1%7D%7D%7B%5Cpartial%20z%5El_j%7D
[z_k_l_1_ _ _sum_limits_p_1_s_l_ _Theta_kp_l_ g_z_p_l_b_k_l]: /images/20220120/86ad348fb8814fcc9370c4bf22658090.png
[frac_partial z_l_1_k_partial z_l_j_ _ _Theta_kj_l_ g_z_l_j]: /images/20220120/59e0a994976849f09e7c7816d6061bde.png
[delta_l_j _ _sum_limits_k_1_s_l_1_ _Theta_kj_l_ _delta_l_1_k g_z_l_j_ _ _sum_limits_k_1_s_l_1_ _Theta_kj_l_ _delta_l_1_k a_j_l _1-a_j_l_ _tag_2]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png%5El_j%20%3D%20%5Csum%5Climits_%7Bk%3D1%7D%5E%7Bs_%7Bl%2B1%7D%7D%20%5CTheta_%7Bkj%7D%5E%7Bl%7D%20%5Cdelta%5E%7Bl%2B1%7D_k%20g'%28z%5El_j%29%20%3D%20%5Csum%5Climits_%7Bk%3D1%7D%5E%7Bs_%7Bl%2B1%7D%7D%20%5CTheta_%7Bkj%7D%5E%7Bl%7D%20%5Cdelta%5E%7Bl%2B1%7D_k%20a_j%5El%20%281-a_j%5El%29%20%5Ctag%7B2%7D
[frac_partial J_Theta_ij_l_ _ _sum_limits_k_1_s_l_1_ _frac_partial J_partial z_k_l_1_ _frac_partial z_k_l_1_partial _Theta_ij_l_ _ _sum_limits_k_1_s_l_ _delta_l_1_k _frac_partial z_k_l_1_partial _Theta_ij_l]: /images/20220120/1c0be74fbe864c5ca20e0bb0915b585e.png
[k_i]: https://juejin.im/equation?tex=k%3Di
[p_j]: /images/20220120/e2710c72f6224eaea519877e246346ab.png
[frac_partial J_Theta_ij_l_ _ g_z_j_l_ _delta_l_1_i _ a_j_l _delta_l_1_i _tag_3]: /images/20220120/8c5b943b3e394abfb5f8377e6186e6e8.png
[l_ i_ j]: https://juejin.im/equation?tex=l%2C%20i%2C%20j
[Delta_ij_l _ 0]: /images/20220120/1d6cf3a5eca545f7a8172e13af8d6b2c.png
[m]: /images/20220120/cd97dddc469b446fbf0ddf854f8b60b8.png
[k]: https://juejin.im/equation?tex=k
[1]: https://juejin.im/equation?tex=1
[a_1 _ x_k]: /images/20220120/59af2d27cfe140f7880ae0c82747eb6e.png
[a_l]: /images/20220120/22d7c3624b2e45058f15c5e5096b1840.png
[delta_L]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png%5EL
[delta_L-1_delta_L-2_..._delta_2]: /images/20220120/e8f0f501aab3478183c2edd468ad0353.png%5E%7BL-1%7D%2C%5Cdelta%5E%7BL-2%7D%2C...%2C%5Cdelta%5E%7B2%7D
[Delta_ij_l]: https://juejin.im/equation?tex=%5CDelta_%7Bij%7D%5El
[Delta_ij_l_Delta_ij_l_a_j_l_delta_i_l_1]: /images/20220120/699302b1e9884c7e80fabcf33a68541c.png
[D_l_ij_ _ _begin_cases_ _dfrac_1_m_Delta_l_ij_ _ _dfrac_lambda_m_Theta_l_ij_ _ _mbox_if _j _neq 0_ _ _dfrac_1_m_Delta_ij_l_ _ _mbox_if _j_0_ _end_cases]: /images/20220120/012463dbab6c480ea94bef785fcbf777.png
[Theta_l _ _Theta_l_ _alpha D_l]: /images/20220120/7258b7fe10cb4e5fb21b97bda2b36fb2.png
[-_delta _leq _Theta_ij_l_ _leq _delta]: /images/20220120/8d2834d6d047491fa775947003dbfb53.png