【Pytorch学习笔记】5.卷积神经网络

柔情只为你懂 2022-09-04 05:56 354阅读 0赞

文章目录

  • 32.卷积神经网络
  • 33.池化层&上/下采样
  • 34.批量正则化
  • 35.经典卷积网络
  • 36.残差网络
  • 37.nn.Module
  • 38.数据增强
  • 39.实战

根据龙良曲Pytorch学习视频整理，视频链接：
【计算机-AI】PyTorch学这个就够了！
(好课推荐)深度学习与PyTorch入门实战——主讲人龙良曲

32.卷积神经网络

基础知识还是得看ng

  1. import torch
  2. import torch.nn as nn
  3. import torch.nn.functional as F
  5. # 卷积层
  6. x = torch.rand(1, 1, 28, 28)
  8. layer = nn.Conv2d(1, 3, kernel_size=3, stride=1, padding=0)
  9. out = layer.forward(x)
  10. print(out.size())   # torch.Size([1, 3, 26, 26])
  12. layer = nn.Conv2d(1, 3, kernel_size=3, stride=1, padding=1)
  13. out = layer.forward(x)
  14. print(out.size())   # torch.Size([1, 3, 28, 28])
  16. layer = nn.Conv2d(1, 3, kernel_size=3, stride=2, padding=1)
  17. out = layer.forward(x)
  18. print(out.size())   # torch.Size([1, 3, 14, 14])
  20. out = layer(x)  # __call__ 推荐使用，不推荐使用forward
  21. print(out.size())   # torch.Size([1, 3, 14, 14])
  23. # print(layer.weight)
  24. print(layer.weight.shape)   # torch.Size([3, 1, 3, 3])
  25. print(layer.bias.shape)     # torch.Size([3])
  27. w = torch.rand(16, 3, 5, 5)
  28. b = torch.rand(16)
  29. x = torch.randn(1, 3, 28, 28)
  30. out = F.conv2d(x, w, b, stride=2, padding=2)
  31. print(out.shape)    # torch.Size([1, 16, 14, 14])

33.池化层&上/下采样

Pooling：feature map变小，与隔行采样（Down sample）不同

  • Max pooling

  • Avg pooling

    池化层

    x = out
    print(x.shape) # torch.Size([1, 16, 14, 14])

    layer = nn.MaxPool2d(2, stride=2)
    out = layer(x)
    print(out.shape) # torch.Size([1, 16, 7, 7])

    out = F.avg_pool2d(x, 2, stride=2)
    print(out.shape) # torch.Size([1, 16, 7, 7])

Up sample

  1. # 上采样
  2. x = out
  3. out = F.interpolate(x, scale_factor=2, mode='nearest')
  4. print(out.shape)    # torch.Size([1, 16, 14, 14])

ReLU

  1. # ReLU
  2. print(x.shape)      # torch.Size([1, 16, 7, 7])
  3. layer = nn.ReLU(inplace=True)
  4. out = layer(x)
  5. print(out.shape)    # torch.Size([1, 16, 7, 7])
  7. out = F.relu(x)
  8. print(out.shape)    # torch.Size([1, 16, 7, 7])

34.批量正则化

1d

  1. import torch
  2. import torch.nn as nn
  4. x = torch.randn(100, 16, 784)
  6. layer = nn.BatchNorm1d(16, momentum=0.1, affine=True)   # affine自动更新beta、gama
  7. # layer.eval() # test时加上
  8. out = layer(x)
  9. print(layer.running_mean)
  10. print(layer.running_var)
  12. for i in range(100):
  13.     out = layer(x)
  14. print(layer.running_mean)
  15. print(layer.running_var)
  16. """ tensor([-3.3207e-04, -2.9735e-04, 8.1316e-04, 9.3234e-05, 1.9049e-04, 6.9931e-04, 2.9378e-04, 3.1153e-05, 3.4325e-04, 3.2283e-04, -2.0425e-04, -4.0346e-04, 1.7246e-04, -1.9482e-04, -1.2086e-04, -8.4132e-04]) tensor([0.9999, 0.9999, 1.0002, 1.0006, 0.9997, 1.0003, 1.0000, 1.0002, 1.0003, 1.0000, 0.9998, 1.0000, 1.0005, 0.9996, 0.9997, 0.9999]) tensor([-0.0033, -0.0030, 0.0081, 0.0009, 0.0019, 0.0070, 0.0029, 0.0003, 0.0034, 0.0032, -0.0020, -0.0040, 0.0017, -0.0019, -0.0012, -0.0084]) tensor([0.9986, 0.9987, 1.0023, 1.0065, 0.9969, 1.0033, 0.9997, 1.0022, 1.0029, 1.0004, 0.9981, 0.9999, 1.0053, 0.9957, 0.9972, 0.9985]) """

layer.running_mean和layer.running_var得到的是全局的均值和方差，不是当前Batch上的，第一次只跑了一个Batch，现在还没有办法直接查看某个Batch上的这两个统计量的值。第一次只进行一次前向传播，在前向传播中来更新均值和方差的值： μ ′ = ( 1 − m ) μ + m μ t = ( 1 − 0.1 ) × 0 + 0.1 × 0.5 = 0.05 \mu ‘= (1-m) \mu + m \mu _t= ( 1 − 0.1 ) × 0 + 0.1 × 0.5 = 0.05 μ′=(1−m)μ+mμt​=(1−0.1)×0+0.1×0.5=0.05
默认动量m = 0.1， μ \mu μ是更新前的均值（初始值为0）， μ t \mu_t μt​是当前batch的平均值。进行多次前向传播，均值和方差就会趋于数据真实分布

注：test时要加上layer.eval()，test不进行反向传播，即 β \beta β和 γ \gamma γ不更新

2d

  1. # 接卷积神经网络实例
  2. print(x.shape)              # torch.Size([1, 16, 7, 7])
  4. layer = nn.BatchNorm2d(16)
  5. out = layer(x)
  6. print(out.shape)            # torch.Size([1, 16, 7, 7])
  7. print(layer.weight.shape)   # torch.Size([16])
  8. print(layer.bias.shape)     # torch.Size([16])
  9. print(vars(layer))          # layer的参数

Advantages

  • Converge faster
  • Better performance
  • Robust
    stable
    larger learning rate

35.经典卷积网络

ImageNet: LeNet-5 → \rightarrow → AlexNet → \rightarrow → VGG → \rightarrow → GoogLeNet → \rightarrow → ResNet → \rightarrow → Inception

36.残差网络

  1. import torch
  2. import torch.nn as nn
  3. import torch.nn.functional as F
  5. class ResBlk(nn.Module):
  6.     def __init__(self, ch_in, ch_out):
  7.         self.conv1 = nn.Conv2d(ch_in, ch_out, kernel_size=3, stride=1, padding=1)
  8.         self.bn1 = nn.BatchNorm2d(ch_out)
  9.         self.conv2 = nn.Conv2d(ch_in, ch_out, kernel_size=3, stride=1, padding=1)
  10.         self.bn2 = nn.BatchNorm2d(ch_out)
  12.         self.extra = nn.Sequential()
  13.         if ch_out != ch_in:
  14.             # [b, ch_in, h, w] => [b, ch_out, h, w]
  15.             self.extra = nn.Sequential(
  16.                 nn.Conv2d(ch_in, ch_out, kernel_size=1, stride=1),
  17.                 nn.BatchNorm2d(ch_out)
  18.             )
  20.     def forward(self, x):
  21.         out = F.relu(self.bn1(self.conv1(x)))
  22.         out = self.bn2(self.conv2(out))
  23.         out = self.extra(x) + out
  24.         return out

37.nn.Module

  • Every Layer is nn.Module
  • nn.Module nested in nn.Module
  1. embed current layers
    Linear ReLU Sigmoid Conv2d ConvTransposed2d Dropout etc.
  2. Container
    net(x)
  3. parameters
  4. modules
    modules: all nodes
    children: direct childrend
  5. to(device)
  6. save and load
    net.load_state_dict(torch.load('ckpt.mdl'))
    torch.save(net.state_dict(), 'ckpt.mdl')
  7. train / test
    net.train()
    net.eval()

  8. implement own layer

    class Flatten(nn.Module):

    1. def __init__(self):
    2.     super(Flatten, self).__init__()
    4. def forward(self, input):
    5.     return input.view(input.size(0), -1)

    class TestNet(nn.Module):

    1. def __init__(self):
    2.     super(TestNet, self).__init__()
    3.     self.net = nn.Sequential(nn.Conv2d(1, 16, stride=1, padding=1),
    4.                              nn.MaxPool2d(2, 2),
    5.                              Flatten(),
    6.                              nn.Linear(1*14*14, 10))
    8. def forward(self, x):
    9.     return self.net(x)

38.数据增强

Limited Data

  • Small network capacity
  • Regularization
  • Data argumentation（will help but not much）

Data argumentation

  • Flip
    transforms.RandomHorizontalFlip() 随机水平翻转
    transforms.RandomVertialFlip() 随机垂直翻转
  • Rotate
    transforms.RandomRotation(15) 随机旋转，不超过15度
    transforms.RandomRotation([90, 180, 270]) 随机在指定角度中旋转
  • Scale
    transforms.Resize([32, 32])
  • Random Move & Crop
    transforms.RandomCrop([28, 28])
  • Noise
  • GAN

39.实战

main.py

  1. import torch
  2. import torch.nn as nn
  3. import torch.optim as optim
  4. from torch.utils.data import DataLoader
  5. from torchvision import transforms, datasets
  6. from Pytorch21_7_29.conv.lenet5 import Lenet5
  7. from Pytorch21_7_29.conv.resnet import ResNet18
  9. def main():
  10.     batch_size = 32
  12.     cifar_train = datasets.CIFAR10('cifar', train=True, transform=transforms.Compose([
  13.         transforms.Resize([32, 32]),
  14.         transforms.ToTensor()
  15.     ]), download=True)
  16.     cifar_train = DataLoader(cifar_train, batch_size=batch_size, shuffle=True)
  18.     cifar_test = datasets.CIFAR10('cifar', train=False, transform=transforms.Compose([
  19.         transforms.Resize([32, 32]),
  20.         transforms.ToTensor(),
  21.         transforms.Normalize(mean=[0.485, 0.456, 0.406],
  22.                              std=[0.229, 0.224, 0.225])
  23.     ]), download=True)
  24.     cifar_test = DataLoader(cifar_test, batch_size=batch_size, shuffle=True)
  26.     x, label = iter(cifar_train).next()
  27.     print("x:", x.shape, 'label:', label.shape) # x: torch.Size([32, 3, 32, 32]) label: torch.Size([32])
  29.     device = torch.device('cuda')
  30.     # model = Lenet5().to(device)
  31.     model = ResNet18().to(device)
  32.     criteon = nn.CrossEntropyLoss()
  33.     optimizer = optim.Adam(model.parameters(), lr=1e-3)
  34.     print(model)
  36.     for epoch in range(1000):
  38.         model.train()
  39.         for batch_idx, (x, label) in enumerate(cifar_train):
  40.             x, label = x.to(device), label.to(device)
  42.             logits = model(x)
  43.             # logits: [b, 10] label: [b]
  44.             loss = criteon(logits, label)   # tensor scalar
  46.             # backward
  47.             optimizer.zero_grad()
  48.             loss.backward()
  49.             optimizer.step()
  51.         #
  52.         print(epoch, loss.item())
  54.         # 测试不需要构建计算图
  55.         model.eval()
  56.         with torch.no_grad():
  57.             # test
  58.             total_correct = 0
  59.             total_num = 0
  60.             for x, label in cifar_test:
  61.                 x, label = x.to(device), label.to(device)
  62.                 logits = model(x)
  63.                 pred = logits.argmax(dim=1)
  64.                 total_correct += torch.eq(pred, label).float().sum().item()
  65.                 total_num += x.size(0)
  67.             acc = total_correct / total_num
  68.             print(epoch, acc)
  70. if __name__ == '__main__':
  71.     main()
  73. """ Lenet5( (conv_unit): Sequential( (0): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1)) (1): AvgPool2d(kernel_size=2, stride=2, padding=0) (2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1)) (3): AvgPool2d(kernel_size=2, stride=2, padding=0) ) (fc_unit): Sequential( (0): Linear(in_features=400, out_features=120, bias=True) (1): ReLU() (2): Linear(in_features=120, out_features=84, bias=True) (3): ReLU() (4): Linear(in_features=84, out_features=10, bias=True) ) (criteon): CrossEntropyLoss() ) ResNet18( (conv1): Sequential( (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(3, 3)) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (blk1): ResBlk( (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (extra): Sequential( (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2)) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (blk2): ResBlk( (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (extra): Sequential( (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2)) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (blk3): ResBlk( (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (extra): Sequential( (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2)) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (blk4): ResBlk( (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (extra): Sequential() ) (outlayer): Linear(in_features=512, out_features=10, bias=True) ) """

lenet5.py

  1. import torch
  2. import torch.nn as nn
  3. import torch.nn.functional as F
  5. class Lenet5(nn.Module):
  6.     """ for cifar10 dataset. """
  7.     def __init__(self):
  8.         super(Lenet5, self).__init__()
  10.         self.conv_unit = nn.Sequential(
  11.             # x:[b, 3, 32, 32] => [b, 6, 28, 28]
  12.             nn.Conv2d(3, 6, kernel_size=5, stride=1, padding=0),
  13.             nn.AvgPool2d(kernel_size=2, stride=2, padding=0),
  14.             # x:[b, 6, 14, 14] => [b, 16, 10, 10]
  15.             nn.Conv2d(6, 16, kernel_size=5, stride=1, padding=0),
  16.             nn.AvgPool2d(kernel_size=2, stride=2, padding=0),
  17.             # x:[b, 16, 5, 5] => [b, 400]
  18.         )
  19.         # flatten
  20.         # fc unit
  21.         self.fc_unit = nn.Sequential(
  22.             nn.Linear(16*5*5, 120),
  23.             nn.ReLU(),
  24.             nn.Linear(120, 84),
  25.             nn.ReLU(),
  26.             nn.Linear(84, 10)
  27.         )
  29.         # self.criteon = nn.MSELoss() # logistics问题使用
  30.         self.criteon = nn.CrossEntropyLoss()    # 一般分类问题使用
  32.     def forward(self, x):
  33.         """ :param x: [b, 3, 32, 32] :return: """
  34.         batch_size = x.size(0)
  35.         x = self.conv_unit(x)
  36.         x = x.view(batch_size, -1)  # -1指16*5*5
  37.         logits = self.fc_unit(x)
  39.         # pred = F.softmax(logits, dim=1) 交叉熵函数内包含了softmax
  40.         # loss = self.criteon(logits, y)
  42.         return logits
  45. def main():
  46.     net = Lenet5()
  47.     tmp = torch.randn(2, 3, 32, 32)
  48.     out = net(tmp)
  49.     print('lenet_out:', out.shape)  # lenet_out: torch.Size([2, 10])
  51. if __name__ == '__main__':
  52.     main()

resnet.py

  1. import torch
  2. import torch.nn as nn
  3. import torch.nn.functional as F
  5. class ResBlk(nn.Module):
  6.     """ resnet block """
  8.     def __init__(self, ch_in, ch_out, stride=1):
  9.         """ :param ch_in: :param ch_out: """
  10.         super(ResBlk, self).__init__()
  12.         # we add stride support for resblk, which is distinct from tutorials.
  13.         self.conv1 = nn.Conv2d(ch_in, ch_out, kernel_size=3, stride=stride, padding=1)
  14.         self.bn1 = nn.BatchNorm2d(ch_out)
  15.         self.conv2 = nn.Conv2d(ch_out, ch_out, kernel_size=3, stride=1, padding=1)
  16.         self.bn2 = nn.BatchNorm2d(ch_out)
  18.         self.extra = nn.Sequential()
  19.         if ch_out != ch_in:
  20.             # [b, ch_in, h, w] => [b, ch_out, h, w]
  21.             self.extra = nn.Sequential(
  22.                 nn.Conv2d(ch_in, ch_out, kernel_size=1, stride=stride),
  23.                 nn.BatchNorm2d(ch_out)
  24.             )
  26.     def forward(self, x):
  27.         """ :param x: [b, ch, h, w] :return: """
  28.         out = F.relu(self.bn1(self.conv1(x)))
  29.         out = self.bn2(self.conv2(out))
  30.         # short cut
  31.         out = self.extra(x) + out
  32.         out = F.relu(out)
  33.         return out
  35. class ResNet18(nn.Module):
  37.     def __init__(self):
  38.         super(ResNet18, self).__init__()
  40.         self.conv1 = nn.Sequential(
  41.             nn.Conv2d(3, 64, kernel_size=3, stride=3, padding=0),
  42.             nn.BatchNorm2d(64)
  43.         )
  45.         # followed 4 blocks
  46.         # [b, 64, h, w] => [b, 128, h, w]
  47.         self.blk1 = ResBlk(64, 128, stride=2)
  48.         # [b, 128, h, w] => [b, 256, h, w]
  49.         self.blk2 = ResBlk(128, 256, stride=2)
  50.         # [b, 256, h, w] => [b, 512, h, w]
  51.         self.blk3 = ResBlk(256, 512, stride=2)
  52.         # [b, 512, h, w] => [b, 1024, h, w]
  53.         self.blk4 = ResBlk(512, 512, stride=2)
  55.         self.outlayer = nn.Linear(512*1*1, 10)
  57.     def forward(self, x):
  58.         """ :param x: :return: """
  59.         x = F.relu(self.conv1(x))
  61.         x = self.blk1(x)
  62.         x = self.blk2(x)
  63.         x = self.blk3(x)
  64.         x = self.blk4(x)
  65.         # print("after_conv:", x.shape) # torch.Size([32, 512, 2, 2])
  66.         # [b, 512, h, w] => [b, 512, 1, 1]
  67.         x = F.adaptive_avg_pool2d(x, [1, 1])
  68.         # print("after_pool:", x.shape) # torch.Size([32, 512, 1, 1])
  69.         x = x.view(x.size(0), -1)
  71.         x = self.outlayer(x)
  73.         return x
  75. def main():
  76.     blk = ResBlk(64, 128, stride=2)
  77.     tmp = torch.randn(2, 64, 32, 32)
  78.     out = blk(tmp)
  79.     print("block:", out.shape)      # torch.Size([2, 128, 16, 16])
  81.     x = torch.randn(2, 3, 32, 32)
  82.     model = ResNet18()
  83.     out = model(x)
  84.     print("resnet:", out.shape)     # torch.Size([2, 10])
  86. if __name__ == '__main__':
  87.     main()

发表评论

表情：
评论列表 （有 0 条评论，354人围观）

还没有评论，来说两句吧...

相关阅读

    相关 pytorch-卷积神经网络基础

    卷积神经网络基础 本节我们介绍卷积神经网络的基础概念，主要是卷积层和池化层，并解释填充、步幅、输入通道和输出通道的含义。 二维卷积层 本节介绍的是最常见的二维卷积

    电玩女神电玩女神/ 0 赞/ 223 阅读