【slowfast复现训练】训练过程制作ava数据集复现 SlowFast Networks for Video Recognition 训练 train

爱被打了一巴掌 2022-09-03 05:24 125阅读 0赞

### 目录 ###

*  [前言][Link 1]
 *  [一，ava相关文件准备][ava]
 *   *  [1.1 空间准备（500G）][1.1 _500G]
     *  [1.2 整体ava文件结构][1.2 _ava]
     *  [1.3 frames文件][1.3 frames]
     *  [1.4 frame\_lists 文件][1.4 frame_lists]
     *  [1.5 annotations 文件][1.5 annotations]
 *  [二，预训练模型][Link 2]
 *  [三，配置文件][Link 3]
 *   *  [3.1 创建新的yaml文件][3.1 _yaml]
     *  [3.2 yaml文件解释][3.2 yaml]
 *  [四，训练][Link 4]

# 前言 #

[b站视频讲解][b]

[slowfast官网][slowfast]

之前写了一个博客，是用slowfast的检测一个自己的视频的demo：  
[【SlowFast复现】SlowFast Networks for Video Recognition复现代码 使用自己的视频进行demo检测][SlowFast_SlowFast Networks for Video Recognition_ _demo]

我又花了2个月时间熟悉slowfast的框架，今天终于把slowfast的训练和数据集这一块跑通了。

这篇博客说一下slowfast的训练过程，这个训练耗时很长，难点体现在训练的数据量巨大，我用了1块GPU，训练了2个视频，估计花了1小时左右，保守估计，要把全部训练完，也就是299个视频，需要150小时左右，也就是6.25天（我没有训练完，只是拿了2个视频来训练，而且用的训练的模型是预训练模型，如果不用这个与训练模型，会花更多时间）

# 一，ava相关文件准备 #

## 1.1 空间准备（500G） ##

首先 在电脑上找一个空间很大的地方，因为ava相关文件要占很大的空间，我花了404G（没有视频，把视频裁剪为图片帧）  
![在这里插入图片描述][20210404195910678.png]

## 1.2 整体ava文件结构 ##

下面是[官网][Link 5]的结构图：

首先，在创建一个文件夹：`ava`，然后在ava文件下创建`frames`、`frame_lists`、`annotations`三个文件夹。

## 1.3 frames文件 ##

在frames文件下，存放的视频的的图片帧，制作过程参考[官网][Link 5]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70]

## 1.4 frame\_lists 文件 ##

这里只存放了两个文件：`train.csv`、`val.csv`  
下载地址：[train.csv][]、[val.csv][]

## 1.5 annotations 文件 ##

这里不要使用官网的默认版本，采用最新的2.2版本，官网有一个下载的链接：[https://dl.fbaipublicfiles.com/pyslowfast/annotation/ava/ava\_annotations.tar][https_dl.fbaipublicfiles.com_pyslowfast_annotation_ava_ava_annotations.tar]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70 1]  
下载后，解压（我是在ubuntu下解压的），使用tree查看其结构：

├── ava_annotations
    │   ├── ava_action_list_v2.1_for_activitynet_2018.pbtxt
    │   ├── ava_action_list_v2.2_for_activitynet_2019.pbtxt
    │   ├── ava_action_list_v2.2.pbtxt
    │   ├── ava_included_timestamps_v2.2.txt
    │   ├── ava_test_excluded_timestamps_v2.1.csv
    │   ├── ava_test_excluded_timestamps_v2.2.csv
    │   ├── ava_test_v2.2.csv
    │   ├── ava_train_excluded_timestamps_v2.1.csv
    │   ├── ava_train_excluded_timestamps_v2.2.csv
    │   ├── ava_train_v2.1.csv
    │   ├── ava_train_v2.2.csv
    │   ├── ava_val_excluded_timestamps_v2.1.csv
    │   ├── ava_val_excluded_timestamps_v2.2.csv
    │   ├── ava_val_v2.1.csv
    │   ├── ava_val_v2.2.csv
    │   ├── person_box_67091280_iou75
    │   │   ├── ava_detection_test_boxes_and_labels.csv
    │   │   ├── ava_detection_train_boxes_and_labels_include_negative.csv
    │   │   ├── ava_detection_train_boxes_and_labels_include_negative_v2.1.csv
    │   │   ├── ava_detection_train_boxes_and_labels_include_negative_v2.2.csv
    │   │   ├── ava_detection_val_boxes_and_labels.csv
    │   │   ├── ava_detection_val_for_training_boxes_and_labels_include_negative.csv
    │   │   └── ava_detection_val_for_training_boxes_and_labels_include_negative_v2.2.csv
    │   ├── person_box_67091280_iou90
    │   │   ├── ava_action_list_v2.1_for_activitynet_2018.pbtxt
    │   │   ├── ava_detection_test_boxes_and_labels.csv
    │   │   ├── ava_detection_train_boxes_and_labels_include_negative.csv
    │   │   ├── ava_detection_train_boxes_and_labels_include_negative_v2.1.csv
    │   │   ├── ava_detection_train_boxes_and_labels_include_negative_v2.2.csv
    │   │   ├── ava_detection_val_boxes_and_labels.csv
    │   │   ├── ava_detection_val_for_training_boxes_and_labels_include_negative.csv
    │   │   ├── ava_detection_val_for_training_boxes_and_labels_include_negative_v2.1.csv
    │   │   ├── ava_detection_val_for_training_boxes_and_labels_include_negative_v2.2.csv
    │   │   ├── ava_train_predicted_boxes.csv
    │   │   ├── ava_train_v2.1.csv
    │   │   ├── ava_val_excluded_timestamps_v2.1.csv
    │   │   ├── ava_val_predicted_boxes.csv -> ava_detection_val_boxes_and_labels.csv
    │   │   ├── ava_val_v2.1.csv
    │   │   ├── test.csv
    │   │   ├── train.csv
    │   │   └── val.csv
    │   ├── test.csv
    │   ├── train.csv
    │   └── val.csv
    └── ava_annotations.tar

将`ava_annotations`下的所有文件复制到`/ava/annotations/`中。

# 二，预训练模型 #

# 三，配置文件 #

## 3.1 创建新的yaml文件 ##

在`/SlowFast/configs/AVA/`下创建一个新的yaml文件：`SLOWFAST_32x2_R50_SHORT3.yaml`，如下图

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70 2]  
然后将下面的代码复制到`SLOWFAST_32x2_R50_SHORT3.yaml`中

TRAIN:
      ENABLE: True
      DATASET: ava
      BATCH_SIZE: 2 #64
      EVAL_PERIOD: 5
      CHECKPOINT_PERIOD: 1
      AUTO_RESUME: True
      CHECKPOINT_FILE_PATH: '/home/lxn/0yangfan/SlowFast/configs/AVA/c2/SLOWFAST_32x2_R101_50_50s.pkl'  #path to pretrain model
      CHECKPOINT_TYPE: caffe2
    DATA:
      NUM_FRAMES: 32
      SAMPLING_RATE: 2
      TRAIN_JITTER_SCALES: [256, 320]
      TRAIN_CROP_SIZE: 224
      TEST_CROP_SIZE: 224
      INPUT_CHANNEL_NUM: [3, 3]
      PATH_TO_DATA_DIR: '/disk6T/ava'
    DETECTION:
      ENABLE: True
      ALIGNED: True
    AVA:
      FRAME_DIR: '/disk6T/ava/frames'
      FRAME_LIST_DIR: '/disk6T/ava/frame_lists'
      ANNOTATION_DIR: '/disk6T/ava/annotations'
      #LABEL_MAP_FILE: 'ava_action_list_v2.1_for_activitynet_2018.pbtxt'
      #0GROUNDTRUTH_FILE: 'ava_val_v2.1.csv'
      #TRAIN_GT_BOX_LISTS: ['ava_train_v2.1.csv']
      DETECTION_SCORE_THRESH: 0.8
      TRAIN_PREDICT_BOX_LISTS: [
        "ava_train_v2.2.csv",
        "person_box_67091280_iou90/ava_detection_train_boxes_and_labels_include_negative_v2.2.csv",
      ]
      #TRAIN_PREDICT_BOX_LISTS: ["ava_train_predicted_boxes.csv"]
      TEST_PREDICT_BOX_LISTS: ["person_box_67091280_iou90/ava_detection_val_boxes_and_labels.csv"]
      #TEST_PREDICT_BOX_LISTS: ["ava_test_predicted_boxes.csv"]
      #EXCLUSION_FILE: "ava_train_excluded_timestamps_v2.1.csv"
      
      
    SLOWFAST:
      ALPHA: 4
      BETA_INV: 8
      FUSION_CONV_CHANNEL_RATIO: 2
      FUSION_KERNEL_SZ: 7
    RESNET:
      ZERO_INIT_FINAL_BN: True
      WIDTH_PER_GROUP: 64
      NUM_GROUPS: 1
      DEPTH: 50
      TRANS_FUNC: bottleneck_transform
      STRIDE_1X1: False
      NUM_BLOCK_TEMP_KERNEL: [[3, 3], [4, 4], [6, 6], [3, 3]]
      SPATIAL_DILATIONS: [[1, 1], [1, 1], [1, 1], [2, 2]]
      SPATIAL_STRIDES: [[1, 1], [2, 2], [2, 2], [1, 1]]
    NONLOCAL:
      LOCATION: [[[], []], [[], []], [[], []], [[], []]]
      GROUP: [[1, 1], [1, 1], [1, 1], [1, 1]]
      INSTANTIATION: dot_product
      POOL: [[[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]]]
    BN:
      USE_PRECISE_STATS: False
      NUM_BATCHES_PRECISE: 200
    SOLVER:
      BASE_LR: 0.1
      LR_POLICY: steps_with_relative_lrs
      STEPS: [0, 10, 15, 20]
      LRS: [1, 0.1, 0.01, 0.001]
      MAX_EPOCH: 20
      MOMENTUM: 0.9
      WEIGHT_DECAY: 1e-7
      WARMUP_EPOCHS: 5.0
      WARMUP_START_LR: 0.000125
      OPTIMIZING_METHOD: sgd
    MODEL:
      NUM_CLASSES: 80
      ARCH: slowfast
      MODEL_NAME: SlowFast
      LOSS_FUNC: bce
      DROPOUT_RATE: 0.5
      HEAD_ACT: sigmoid
    TEST:
      ENABLE: False
      DATASET: ava
      BATCH_SIZE: 8
    DATA_LOADER:
      NUM_WORKERS: 2
      PIN_MEMORY: True
    NUM_GPUS: 1
    NUM_SHARDS: 1
    RNG_SEED: 0
    OUTPUT_DIR: .

## 3.2 yaml文件解释 ##

1.  TRAIN  
    1.1. ENABLE: True。这里将TRAIN设置为TRUE，同样的，也要TEST.ENABLE设置为False（我们只需要训练的过程）  
    1.2 BATCH\_SIZE: 2 \#64.这里batch\_size是由于我的电脑显存不够，只能设置为2，如果大家的显存够大，可以把这个batch\_size设置的大一些。  
    1.3 CHECKPOINT\_FILE\_PATH: ‘/home/lxn/0yangfan/SlowFast/configs/AVA/c2/SLOWFAST\_32x2\_R101\_50\_50s.pkl’ 这里放的是预训练模型的位置
2.  DATA  
    2.1 PATH\_TO\_DATA\_DIR: ‘/disk6T/ava’ 这里是ava文件的位置

其他也比较简单，大家就自己理解了。

# 四，训练 #

python tools/run_net.py --cfg configs/AVA/SLOWFAST_32x2_R50_SHORT3.yaml

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70 3]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70 4]

[Link 1]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#_1
[ava]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#ava_15
[1.1 _500G]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#11_500G_16
[1.2 _ava]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#12_ava_19
[1.3 frames]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#13_frames_44
[1.4 frame_lists]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#14_frame_lists__48
[1.5 annotations]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#15_annotations__52
[Link 2]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#_109
[Link 3]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#_111
[3.1 _yaml]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#31_yaml_112
[3.2 yaml]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#32_yaml_214
[Link 4]: https://blog.csdn.net/WhiffeYF/article/details/115431794?spm=1001.2014.3001.5501#_225
[b]: https://www.bilibili.com/video/BV1WB4y1w7Mk
[slowfast]: https://github.com/facebookresearch/SlowFast
[SlowFast_SlowFast Networks for Video Recognition_ _demo]: https://blog.csdn.net/WhiffeYF/article/details/113527759?spm=1001.2014.3001.5501
[20210404195910678.png]: /images/20220829/a9c7150dc613452c94639d804d9390c7.png
[Link 5]: https://github.com/facebookresearch/SlowFast/blob/master/slowfast/datasets/DATASET.md
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70]: /images/20220829/44d3248b6d3949cfac8b97d554c8c5e0.png
[train.csv]: https://dl.fbaipublicfiles.com/video-long-term-feature-banks/data/ava/frame_lists/train.csv
[val.csv]: https://dl.fbaipublicfiles.com/video-long-term-feature-banks/data/ava/frame_lists/val.csv
[https_dl.fbaipublicfiles.com_pyslowfast_annotation_ava_ava_annotations.tar]: https://dl.fbaipublicfiles.com/pyslowfast/annotation/ava/ava_annotations.tar
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70 1]: /images/20220829/89f331fd223a44c8a52b5d5378303b81.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70 2]: /images/20220829/7f1ef46150ae44c2a945b7cda3a2c192.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70 3]: /images/20220829/d731dbb494bc42d080e9156c52288816.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1doaWZmZVlG_size_16_color_FFFFFF_t_70 4]: /images/20220829/11c71797f592476e8ab28d7d3cdb514f.png