基于深度学习的３D骨架动作识别

梦里梦外; 2022-05-20 09:41 461阅读 0赞

关于２Ｄ视频动作识别，请移步这两篇[博客1][1],[博客2][2]

３Ｄ骨架动作识别，现在主要有两种方法，一是用LSTM进行时序上的记忆，二是将骨架坐标转换成特殊的图片，用卷积网络进行特征提取以及记忆，三是将LSTM和CNN网络的结果进行融合  
目前使用的最大数据集是NTU-RGBD，后来又出现了一个PKU-MMD，但这个数据集是一段视频包含多段动作，主要是用来做detection的。  
这个领域的开源代码非常少，微软有几篇不错的文章，但都不开源，初学者表示很是头疼。以下每个分类按时间发布先后进行整理。也有可能有没看过的文章，本文大概介绍一些比较经典的用来做baseline的文章。  
本问所讲的所有paper，输入信息都只有３ｄ骨架坐标。

### 基于lstm的３d骨架动作识别 ###

因为骨架信息是３Ｄ坐标，两帧之间的变化非常明显，用lstm效果非常显著。基于深度学习的方法最开始也是从ＲＮＮ开始入手。然后在肢体各个部分的特征融合方面下功夫。  
[View Adaptive Recurrent Neural Networks for High Performance　Human Action Recognition from Skeleton Data][View Adaptive Recurrent Neural Networks for High Performance_Human Action Recognition from Skeleton Data]　2017/4/8　ICCV2017  
这是我最深入研究的一个文章，文章思路比较巧妙，用两个子网络回归出旋转平移矩阵的参数，将骨架坐标旋转到一个合适的视角，再送入主网络进行动作识别。  
[Modeling Temporal Dynamics and Spatial Configurations of Actions UsingTwo-Stream Recurrent Neural Networks][] CVPR2017  
双流ＲＮＮ,  
![这里写图片描述][70]  
NTU-RGBD: CS 71.3% , CV 79.5%  
[Skeleton Based Human Action Recognition with Global Context-Aware Attention LSTM Networks][] CVPR2017  
对注意力模型进行了优化，将初始背景信息送给每一个step,因为对于lstm而言，每帧记忆的隐藏信息还是非常local.  
[An End-to-End Spatio-Temporal Attention Model for Human Action Recognition　from Skeleton Data][An End-to-End Spatio-Temporal Attention Model for Human Action Recognition_from Skeleton Data]　2016/11  
时空注意力模型，引入了注意力机制，缺点是训练过程比较复杂

### 基于CNN的３d骨架动作识别 ###

[Co-occurrence Feature Learning from Skeleton Data for Action Recognition and  
Detection with Hierarchical Aggregation][Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation]IJCAI 2018  
来自海康威视的论文，应该是目前最新的研究成果，精度也是state-of-art,参考[博客][Link 1]  
[Deep Progressive Reinforcement Learning for Skeleton-based Action Recognition][] CVPR2018　  
来自清华的增强学习  
[A New Representation of Skeleton Sequences for 3D Action Recognition][] CVPR2017  
参考[博客][Link 2]  
[Investigation of Different Skeleton Features for CNN-based 3D Action Recognition][] 2017/5  
将节点之间的距离变成图片，将线和线之间的夹角变成图片，再送入CNN网络提取特征

### CNN和lstm的融合 ###

[Skeleton-based Action Recognition Using LSTM and CNN][]　ICMEW 2017 2017/7  
该文章使用了７个LSTM和３个CNN  
NTU-RGBD: CS 82.89%, CV 90.1%

[1]: https://blog.csdn.net/whfshuaisi/article/details/79116246
[2]: https://blog.csdn.net/whfshuaisi/article/details/79116265
[View Adaptive Recurrent Neural Networks for High Performance_Human Action Recognition from Skeleton Data]: https://arxiv.org/pdf/1703.08274v2.pdf
[Modeling Temporal Dynamics and Spatial Configurations of Actions UsingTwo-Stream Recurrent Neural Networks]: http://openaccess.thecvf.com/content_cvpr_2017/papers/Wang_Modeling_Temporal_Dynamics_CVPR_2017_paper.pdf
[70]: /images/20220520/94a41bad8dc24fc6842c8cfbef6aab7a.png
[Skeleton Based Human Action Recognition with Global Context-Aware Attention LSTM Networks]: https://arxiv.org/pdf/1707.05740v3.pdf
[An End-to-End Spatio-Temporal Attention Model for Human Action Recognition_from Skeleton Data]: https://arxiv.org/pdf/1611.06067v1.pdf
[Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation]: https://arxiv.org/pdf/1804.06055.pdf
[Link 1]: https://zhuanlan.zhihu.com/p/36007896
[Deep Progressive Reinforcement Learning for Skeleton-based Action Recognition]: http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/1736.pdf
[A New Representation of Skeleton Sequences for 3D Action Recognition]: http://openaccess.thecvf.com/content_cvpr_2017/papers/Ke_A_New_Representation_CVPR_2017_paper.pdf
[Link 2]: https://blog.csdn.net/u013608336/article/details/78740343
[Investigation of Different Skeleton Features for CNN-based 3D Action Recognition]: https://arxiv.org/pdf/1705.00835v1.pdf
[Skeleton-based Action Recognition Using LSTM and CNN]: https://arxiv.org/pdf/1707.02356v1.pdf