权重衰减与L2范数正则化

桃扇骨 2023-01-22 04:54 85阅读 0赞

正则化就是在损失函数后加上一个自定义的正则项，假设我们用线性模型的损失函数均方差为例  
 l o s s = 1 n ( ∑ 1 n ( w 1 x 1 + w 2 x 2 + b − y ^ ) 2 + λ ∣ ∣ w ∣ ∣ 2 ) loss = \\frac 1 n\\big(\\sum\_1^n(w\_1x\_1+w\_2x\_2+b-\\hat y)^2 + \\lambda ||w||^2 \\big) loss=n1(∑1n(w1x1\+w2x2\+b−y^)2\+λ∣∣w∣∣2)  
 ∣ ∣ w ∣ ∣ 2 = 1 2 ( w 1 2 + w 2 2 ) ||w||^2 = \\frac 1 2 (w\_1^2 + w\_2^2) ∣∣w∣∣2=21(w12\+w22)  
这里的  λ ∣ ∣ w ∣ ∣ 2 \\lambda ||w||^2 λ∣∣w∣∣2就是L2范数正则项，将它加入到损失函数中，，梯度下降时模型会倾向于选择参数较小的模型，这样使得一个较为复杂的model的function set变小，即model的弹性变小，在一定程度上可以缓解过拟合。

L2范数正则化其实就是权重衰减，为什么这么说呢？  
加上L2范数正则项的loss对各参数进行求导  
 ∂ l o s s w 1 = 2 n ∑ 1 n ( w 1 x 1 + w 2 x 2 + b − y ^ ) ∗ x 1 + λ w 1 \\frac \{\\partial loss\} \{w\_1\} = \\frac 2 n\\sum\_1^n(w\_1x\_1+w\_2x\_2+b-\\hat y) \* x\_1 + \\lambda w\_1 w1∂loss=n2∑1n(w1x1\+w2x2\+b−y^)∗x1\+λw1  
 w 1 ∗ = ( 1 − λ ) w 1 − g r a d w\_1^\* = (1 - \\lambda)w\_1 - grad w1∗=(1−λ)w1−grad  
每次参数都会先乘上一个小于1的系数，然后再减去原来的梯度，所以叫权重衰减

**下面进行一个高维线性实验**  
假设我们的真实方程是：  
![在这里插入图片描述][20210517162758452.png]  
假设feature数200，训练样本和测试样本各20个

## 模拟数据集 ##

num_train,num_test = 10,10
    num_features = 200
    true_w = torch.ones((num_features,1),dtype=torch.float32) * 0.01
    true_b = torch.tensor(0.5)
    samples = torch.normal(0,1,(num_train+num_test,num_features))
    noise = torch.normal(0,0.01,(num_train+num_test,1))
    labels = samples.matmul(true_w) + true_b + noise
    train_samples, train_labels= samples[:num_train],labels[:num_train]
    test_samples, test_labels = samples[num_train:],labels[num_train:]

## 定义带正则项的loss function ##

def loss_function(predict,label,w,lambd):
        loss = (predict - label) ** 2
        loss = loss.mean() + lambd * (w**2).mean()
        return loss

## 画图的方法 ##

def semilogy(x_val,y_val,x_label,y_label,x2_val,y2_val,legend):
        plt.figure(figsize=(3,3))
        plt.xlabel(x_label)
        plt.ylabel(y_label)
        plt.semilogy(x_val,y_val)
        if x2_val and y2_val:
            plt.semilogy(x2_val,y2_val)
            plt.legend(legend)
        plt.show()

## 拟合和画图 ##

def fit_and_plot(train_samples,train_labels,test_samples,test_labels,num_epoch,lambd):
        w = torch.normal(0,1,(train_samples.shape[-1],1),requires_grad=True)
        b = torch.tensor(0.,requires_grad=True)
        optimizer = torch.optim.Adam([w,b],lr=0.05)
        train_loss = []
        test_loss = []
        for epoch in range(num_epoch):
            predict = train_samples.matmul(w) + b
            epoch_train_loss = loss_function(predict,train_labels,w,lambd)
            optimizer.zero_grad()
            epoch_train_loss.backward()
            optimizer.step()
            test_predict = test_sapmles.matmul(w) + b
            epoch_test_loss = loss_function(test_predict,test_labels,w,lambd)
            train_loss.append(epoch_train_loss.item())
            test_loss.append(epoch_test_loss.item())
        semilogy(range(1,num_epoch+1),train_loss,'epoch','loss',range(1,num_epoch+1),test_loss,['train','test'])

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTUyNjIy_size_16_color_FFFFFF_t_70]  
可以发现加了正则项的模型，在测试集上的loss确实下降了

[20210517162758452.png]: /images/20221020/89b080a410a94b2ca44069c42c00c18c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTUyNjIy_size_16_color_FFFFFF_t_70]: /images/20221020/5f7885cd241143c1ab5b349330c5f093.png