另辟蹊径——误差逆传播算法识别MNIST手写数字

左手的ㄟ右手 2024-03-30 15:36 58阅读 0赞

MNIST数据集(Mixed National Institute of Standards and Technology database)是美国国家标准与技术研究院收集整理的大型手写数字数据库,包含60,000个示例的训练集以及10,000个示例的测试集。

![3560e5e2daa07cc5e53ce2c89602d719.jpeg][]

每个数字都是28\*28的矩阵，总计是784个数据，我们将这784个数据“拉成”一维矩阵，将其作为多层感知机的“x”。

![91ec634d6cbe49db9242b5d089433988.png][]

误差逆传播算法总体上来看就是由众多![y=kx+b][y_kx_b]和sigmoid函数组成的集合，在算法中，阈值和连接权（权重）使用随即初始值，例如在pytorch中我们可以使用以下方法实现

num_inputs = 784
    num_outputs = 10
    num_hiddens = 256
    
    w1 = nn.Parameter(torch.randn(num_inputs,num_hiddens,requires_grad=True)*0.1)
    b1 = nn.Parameter(torch.zeros(num_hiddens,requires_grad=True))
    w2 = nn.Parameter(torch.randn(num_hiddens,num_outputs,requires_grad=True)*0.1)
    b2 = nn.Parameter(torch.zeros(num_outputs,requires_grad=True))

假定训练集为![\\left ( \\sum\_\{i=1\}^\{784\}x\_\{ik\},y\_\{k\} \\right )][left _ _sum_i_1_784_x_ik_y_k_ _right]，我们设一层隐层，一层输出层，隐层中有256个神经元，输出层有10个神经元，分别是0，1，2，3，4，5，6，7，8，9。那么隐层的输入应为

![\\alpha \_\{h\} = \\sum\_\{i=1\}^\{784\}v\_\{ih\}x\_\{i\}+b\_\{1h\}][alpha _h_ _ _sum_i_1_784_v_ih_x_i_b_1h]

输出层神经元的输入为

![\\beta \\widehat\{\}\_\{j\} = \\sum\_\{i=1\}^\{256\}\\omega \_\{ij\}x\_\{i\}][beta _widehat_j_ _ _sum_i_1_256_omega _ij_x_i]

这里还需要使用sigmoid函数对每一个y进行“压缩”

![y\\widehat\{\}\_\{j\}^\{k\} = sigmoid\\left ( \\beta \_\{j\}-\\theta \_\{j\}\\right )][y_widehat_j_k_ _ sigmoid_left _ _beta _j_-_theta _j_right]

则网络在![\\left ( \\sum\_\{i=1\}^\{784\}x\_\{ik\},y\_\{k\} \\right )][left _ _sum_i_1_784_x_ik_y_k_ _right]的均方误差为

![E\_\{k\} = \\frac\{1\}\{2\}\\sum\_\{j=1\}^\{l\}\\left ( y\\widehat\{\}\_\{j\}^\{k\}-y\_\{j\}^\{k\} \\right )][E_k_ _ _frac_1_2_sum_j_1_l_left _ y_widehat_j_k_-y_j_k_ _right]

误差逆传播算法基于梯度下降策略，以目标的负梯度方向对参数进行调整,任意参数v的更新估计式为

![v\\leftarrow v+\\Delta v][v_leftarrow v_Delta v]

对于上述均方误差，给定学习率![\\eta][eta]，有

![\\Delta \\omega\_\{hj\} = -\\eta \\frac\{\\delta E\_\{k\}\}\{\\delta \\omega \_\{hj\}\}][Delta _omega_hj_ _ -_eta _frac_delta E_k_delta _omega _hj]

误差逆传播算法是一个迭代学习算法，类似上述公式，将其余值的![\\Delta][Delta]求出，并结合更新估计式对参数进行更新估计。需注意的是，误差逆传播算法的目标是要最小化训练集D上的累计误差

![E=\\frac\{1\}\{m\}\\sum\_\{k=1\}^\{m\}E\_\{k\}][E_frac_1_m_sum_k_1_m_E_k]

import image_train
    import torch
    from torch import nn
    from d2l import torch as d2l
    
    train_x = torch.tensor(image_train.images).float()
    train_y = torch.tensor(image_train.labels).float()
    
    batch_size = 60000
    num_inputs = 784
    num_outputs = 10
    num_hiddens = 256
    
    w1 = nn.Parameter(torch.randn(num_inputs,num_hiddens,requires_grad=True)*0.1)
    b1 = nn.Parameter(torch.zeros(num_hiddens,requires_grad=True))
    w2 = nn.Parameter(torch.randn(num_hiddens,num_outputs,requires_grad=True)*0.1)
    b2 = nn.Parameter(torch.zeros(num_outputs,requires_grad=True))
    
    parms = [w1,b1,w2,b2]
    
    def sigmoid(x):
        y = torch.sigmoid(x)
        return y
    
    index = 0
    for i in range(20001):
        h = sigmoid(train_x[i] @ w1 + b1)
        y = sigmoid(h @ w2 + b2)
        g = y * (1 - y) * (train_y[i] - y)
        index=g
        gg = g.reshape(1,-1).T
        hh = h.reshape(1,-1)
        delta_w2 = 0.1*(gg@hh).T
        delta_b2 = -0.1*g
        w2 = w2+delta_w2
        b2 = b2+delta_b2
        e = hh*(1-hh)*(w2@gg.reshape((-1,1))).T#1,256
        xx = train_x[i].reshape(1,-1)
        delta_w1 = 0.1*xx.T@e
        delta_b1 = -0.1*e
        w1 = w1+delta_w1
        b1 = b1+delta_b1
    
    
    a = index.tolist()[0]
    b = train_y[20000].tolist()
    print('预测',a.index(max(a)))
    print('实际',b.index(max(b)))

[3560e5e2daa07cc5e53ce2c89602d719.jpeg]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/ecb82414d8fc4f648598e034993eb5bc.jpeg
[91ec634d6cbe49db9242b5d089433988.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/a293ce21ba03455481f4a587c3a3ba5d.png
[y_kx_b]: https://latex.csdn.net/eq?y%3Dkx&plus;b
[left _ _sum_i_1_784_x_ik_y_k_ _right]: https://latex.csdn.net/eq?%5Cleft%20%28%20%5Csum_%7Bi%3D1%7D%5E%7B784%7Dx_%7Bik%7D%2Cy_%7Bk%7D%20%5Cright%20%29
[alpha _h_ _ _sum_i_1_784_v_ih_x_i_b_1h]: https://latex.csdn.net/eq?%5Calpha%20_%7Bh%7D%20%3D%20%5Csum_%7Bi%3D1%7D%5E%7B784%7Dv_%7Bih%7Dx_%7Bi%7D&plus;b_%7B1h%7D
[beta _widehat_j_ _ _sum_i_1_256_omega _ij_x_i]: https://latex.csdn.net/eq?%5Cbeta%20%5Cwidehat%7B%7D_%7Bj%7D%20%3D%20%5Csum_%7Bi%3D1%7D%5E%7B256%7D%5Comega%20_%7Bij%7Dx_%7Bi%7D
[y_widehat_j_k_ _ sigmoid_left _ _beta _j_-_theta _j_right]: https://latex.csdn.net/eq?y%5Cwidehat%7B%7D_%7Bj%7D%5E%7Bk%7D%20%3D%20sigmoid%5Cleft%20%28%20%5Cbeta%20_%7Bj%7D-%5Ctheta%20_%7Bj%7D%5Cright%20%29
[E_k_ _ _frac_1_2_sum_j_1_l_left _ y_widehat_j_k_-y_j_k_ _right]: https://latex.csdn.net/eq?E_%7Bk%7D%20%3D%20%5Cfrac%7B1%7D%7B2%7D%5Csum_%7Bj%3D1%7D%5E%7Bl%7D%5Cleft%20%28%20y%5Cwidehat%7B%7D_%7Bj%7D%5E%7Bk%7D-y_%7Bj%7D%5E%7Bk%7D%20%5Cright%20%29
[v_leftarrow v_Delta v]: https://latex.csdn.net/eq?v%5Cleftarrow%20v&plus;%5CDelta%20v
[eta]: https://latex.csdn.net/eq?%5Ceta
[Delta _omega_hj_ _ -_eta _frac_delta E_k_delta _omega _hj]: https://latex.csdn.net/eq?%5CDelta%20%5Comega_%7Bhj%7D%20%3D%20-%5Ceta%20%5Cfrac%7B%5Cdelta%20E_%7Bk%7D%7D%7B%5Cdelta%20%5Comega%20_%7Bhj%7D%7D
[Delta]: https://latex.csdn.net/eq?%5CDelta
[E_frac_1_m_sum_k_1_m_E_k]: https://latex.csdn.net/eq?E%3D%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bk%3D1%7D%5E%7Bm%7DE_%7Bk%7D