(fast-reid)torch error:DataLoader worker (pid(s) 26462) exited unexpectedly

刺骨的言语ヽ痛彻心扉 2023-02-25 08:44 30阅读 0赞

## 1.问题 ##

之前的时候已经配置好了fast-reid的环境，隔了以后再次运行这个，发现出现缓存不足的问题：

File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/reductions.py", line 315, in reduce_storage
        fd, size = storage._share_fd_()
    RuntimeError: unable to write to file </torch_27458_75757216>
      0%|                                                                                                                                         | 0/76 [00:27<?, ?it/s]
    Traceback (most recent call last):
    RuntimeError: DataLoader worker (pid 26462) is killed by signal: Bus error.

## 2.分析 ##

设置更低的num\_work。发现还是出问题。

由于pytorch会写东西保存到缓存，但是磁盘不足的问题。

github issue上给出了解决方案[https://github.com/huaweicloud/dls-example/issues/26\#issuecomment-411990039][https_github.com_huaweicloud_dls-example_issues_26_issuecomment-411990039]：

将下面代码放到要运行的py文件import导入模型之后：

import sys
    import torch
    from torch.utils.data import dataloader
    from torch.multiprocessing import reductions
    from multiprocessing.reduction import ForkingPickler
    
    default_collate_func = dataloader.default_collate
    
    
    def default_collate_override(batch):
      dataloader._use_shared_memory = False
      return default_collate_func(batch)
    
    setattr(dataloader, 'default_collate', default_collate_override)
    
    for t in torch._storage_classes:
      if sys.version_info[0] == 2:
        if t in ForkingPickler.dispatch:
            del ForkingPickler.dispatch[t]
      else:
        if t in ForkingPickler._extra_reducers:
            del ForkingPickler._extra_reducers[t]

# 3.可能的解决方法 #

[https://github.com/pytorch/pytorch/issues/5040][https_github.com_pytorch_pytorch_issues_5040]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70][]

## 参考 ##

[1.huaweicloud/dls-example/issues/26][https_github.com_huaweicloud_dls-example_issues_26_issuecomment-411990039]

[https_github.com_huaweicloud_dls-example_issues_26_issuecomment-411990039]: https://github.com/huaweicloud/dls-example/issues/26#issuecomment-411990039
[https_github.com_pytorch_pytorch_issues_5040]: https://github.com/pytorch/pytorch/issues/5040
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM1OTc1NDQ3_size_16_color_FFFFFF_t_70]: /images/20230209/9c7ca77ec27c4a4597aa52b6d90aaf66.png