PyTorch 数据集随机值的完美实践

青旅半醒 2022-08-28 03:50 203阅读 0赞

点上方**计算机视觉联盟**获取更多干货

仅作学术分享，不代表本公众号立场，侵权联系删除

转载于：作者 | Elvanth@知乎  
来源 | https://zhuanlan.zhihu.com/p/377155682  
编辑 | 极市平台

**985人工智能博士笔记推荐**

**[周志华《机器学习》手推笔记正式开源！附pdf下载链接，Github2500星！][pdf_Github2500]**

一个快捷的解决方案：

def worker_init_fn(worker_id):
        worker_seed = torch.initial_seed() % 2**32
        np.random.seed(worker_seed)
        random.seed(worker_seed)
    
    ds = DataLoader(ds, 10, shuffle=False, num_workers=4, worker_init_fn=worker_init_fn)

## 01 关于pytorch数据集随机种子的基本认识 ##

在pytorch中random、torch.random等随机值产生方法一般没有问题，只有少数工人运行也可以保障其不同的最终值.

np.random.seed 会出现问题的原因是，当多处理采用 fork 方式产生子进程时，numpy 不会对不同的子进程产生不同的随机值.

换言之，当没有多处理使用时，numpy 不会出现随机种子的不同的问题；实验代码的可复现性要求**一个是工人种子** ,即工人内包括numpy，random，torch.random所有的随机表现；**另一个是Base** ,即程序运行后的初始随机值，其可以通过以下两种方式产生

1.  torch.manual\_seed(base\_seed)
2.  由特定的seed generator设置

generator = torch. Generator()
    g.manual_seed(base_seed)
    DataLoader(dataset, ..., generator=generator)

使用spawn模式可以斩断以上所有烦恼.

## 02 直接在网上搜这个问题会得到什么答案 ##

参考很多的解决方案时，往往会提出以下功能：

defworker_init_fn(worker_id):
        np.random.seed(np.random.get_state()[1][0] + worker_id)

让我们看看它的输出结果：  
（第0，3列是索引，第1，4列是np.random的结果，第2，5列是random.randint的结果）

epoch 0
    tensor([[    0,  5125, 13588,     0, 15905, 23182],
            [    1,  7204, 19825,     0, 13653, 25225]])
    tensor([[    2,  1709, 11504,     0, 12842, 23238],
            [    3,  5715, 14058,     0, 15236, 28033]])
    tensor([[    4,  1062, 11239,     0, 10142, 29869],
            [    5,  6574, 15672,     0, 19623, 25600]])
    ============================================================
    epoch 1
    tensor([[    0,  5125, 18134,     0, 15905, 28990],
            [    1,  7204, 13206,     0, 13653, 25106]])
    tensor([[    2,  1709, 15512,     0, 12842, 29703],
            [    3,  5715, 14201,     0, 15236, 27696]])
    tensor([[    4,  1062, 13994,     0, 10142, 23411],
            [    5,  6574, 18532,     0, 19623, 21744]])
    ============================================================

假设上述方案对一个时代内可以防止不同的工人出现随机值相同的情况，但**不同的时代之间，其最终的随机种子仍然是不变的。**

## 03 那应该如何解决 ##

### 来自pytorch官方的解决方案： ###

https://github.com/pytorch/pytorch/pull/56488\#issuecomment-825128350

def worker_init_fn(worker_id):
        worker_seed = torch.initial_seed() % 2**32
        np.random.seed(worker_seed)
        random.seed(worker_seed)
     
    ds = DataLoader(ds, 10, shuffle=False, num_workers=4, worker_init_fn=worker_init_fn)

### 来自numpy.random原作者的解决方案： ###

https://github.com/pytorch/pytorch/issues/5059\#issuecomment-817392562

def worker_init_fn(id):
        process_seed = torch.initial_seed()
        # Back out the base_seed so we can use all the bits.
        base_seed = process_seed - id
        ss = np.random.SeedSequence([id, base_seed])
        # More than 128 bits (4 32-bit words) would be overkill.
        np.random.seed(ss.generate_state(4))
     
    ds = DataLoader(ds, 10, shuffle=False, num_workers=4, worker_init_fn=worker_init_fn)

### 一个更简单但不保证正确性的解决方案： ###

def worker_init_fn(worker_id):
        np.random.seed((worker_id + torch.initial_seed()) % np.iinfo(np.int32).max)
    
    ds = DataLoader(ds, 10, shuffle=False, num_workers=4, worker_init_fn=worker_init_fn)

### 04 附上可运行的完整文件 ###

import numpy as np
    import random
    import torch
    
    # np.random.seed(0)
    
    class Transform(object):
        def __init__(self):
            pass
    
        def __call__(self, item = None):
            return [np.random.randint(10000, 20000), random.randint(20000,30000)]
    
    class RandomDataset(object):
        def __init__(self):
            pass
    
        def __getitem__(self, ind):
            item = [ind, np.random.randint(1, 10000), random.randint(10000, 20000), 0]
            tsfm =Transform()(item)
            return np.array(item + tsfm)
        def __len__(self):
            return20
    
    from torch.utils.data import DataLoader
    
    def worker_init_fn(worker_id):
        np.random.seed(np.random.get_state()[1][0] + worker_id)
    
    ds = RandomDataset()
    ds = DataLoader(ds, 10, shuffle=False, num_workers=4, worker_init_fn=worker_init_fn)
    
    for epoch in range(2):
        print("epoch {}".format(epoch))
        np.random.seed()
        for batch in ds:
            print(batch)

**-------------------**

***END***

***--------------------***

我是[王博Kings][Kings]，985AI博士，华为云专家、CSDN博客专家（人工智能领域优质作者）。单个AI开源项目现在已经获得了2100+标星。现在在做AI相关内容，欢迎一起交流学习、生活各方面的问题，一起加油进步！

我们微信交流群涵盖以下方向（但并不局限于以下内容）：人工智能，计算机视觉，自然语言处理，目标检测，语义分割，自动驾驶，GAN，强化学习，SLAM，人脸检测，最新算法，最新论文，OpenCV，TensorFlow，PyTorch，开源框架，学习方法...

这是我的私人微信，位置有限，一起进步！

**![4ffa03cc8973119da1914d17e54f88c9.png][]**

**王博的公众号，欢迎关注，干货多多**

**手推笔记：**

**增长见识：**

**其他学习笔记：**

![a1511984fdca8d782e354efee9e17b58.gif][]

**点分享**

![cbe6d2b6e3dcd63935c71ab7da789d79.gif][]

**点收藏**

![7c5d4d183d6b41ad202955f2aff82af9.gif][]

**点点赞**

![695156925f204a3a6ef126d0fee8caea.gif][]

**点在看**

[pdf_Github2500]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486763&idx=1&sn=da878ccafba79932be8eeca955697477&chksm=fcb71194cbc0988259f32b0b7a77fe7830c477c9c9a4fcb00f2e366c97d776574c2bc949e993&scene=21#wechat_redirect
[Kings]: https://blog.csdn.net/Sophia_11/article/details/114274846
[4ffa03cc8973119da1914d17e54f88c9.png]: /images/20220828/f81e6b2ca1e24be1ab97f158ebdb24eb.png
[Link 1]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486235&idx=1&sn=60774f6e71af3c1df5fcdaf37407c1ce&chksm=fcb717a4cbc09eb2915aa8504be858f681e5cc4ea95cc091335f021b30b566f2f16e03ae0739&scene=21#wechat_redirect
[Link 2]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486809&idx=2&sn=1db1d3e29268334b78faf1b981030974&chksm=fcb711e6cbc098f0db247fdf9739a4e82aaada8231d5b3c6c79b1c19e79d81e3216a7e7476d3&scene=21#wechat_redirect
[Link 3]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486859&idx=1&sn=9ee1c93b8ca69d71ea1db16e022f2691&chksm=fcb71134cbc09822e382eaeb834f9ee66f78ea78636494cd22bcb0493b4e2ae7ef57fe99fe58&scene=21#wechat_redirect
[Link 4]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247486891&idx=1&sn=48f35f7bcaa7dd559288231c3ac07e30&chksm=fcb71114cbc09802e6d99fe3cd5a41d6da03b1e9eb5442b01b84d9c9edcd49cad0bb50a1309c&scene=21#wechat_redirect
[Link 5]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247487004&idx=1&sn=b5f13ba410cee5730d1e3784b7fc4c9b&chksm=fcb712a3cbc09bb5aa3e4f16254cd52a72a3e5393370bc5c1ca78682161c74ad81443e01135d&scene=21#wechat_redirect
[Link 6]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247487301&idx=1&sn=a5220ac4604bfdb15ee2eb52357a19c1&chksm=fcb713facbc09aecde8ec575fe96d7ef2b406b4bb377c63e197cdeaf04cba14e76b8f75c30e4&scene=21#wechat_redirect
[Link 7]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247487435&idx=2&sn=d3d9a9ecd93f6defe865da36aede1777&chksm=fcb71374cbc09a62446ef8f03e22dcd2b1092169ee0a472a7d3f0e1d6209300943b03d389e76&scene=21#wechat_redirect
[Link 8]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247488226&idx=1&sn=6694a44c5bc8d8207e8348cfbb141297&chksm=fcb70e5dcbc0874b355226a7ef07f0d2e315b13df95c3df2812a70b4eb6ab2d68184e1c3cc86&scene=21#wechat_redirect
[Link 9]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247488404&idx=2&sn=36b6c46669b1c5f1c0922f3c476b2d85&chksm=fcb70f2bcbc0863d86fdf807a48cd919d75556299c4b911602a052cd92d31cc57fbae1600fe2&scene=21#wechat_redirect
[Link 10]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247488468&idx=2&sn=7071a2a562afa337e7b46b490aa44090&chksm=fcb70f6bcbc0867dcd4cd5a16a7b5ce3b88bb10e9af6165797ffe10f7cca8b9d3a0eb365774f&scene=21#wechat_redirect
[Link 11]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247488759&idx=2&sn=91be2e59ada4107851289c68a64bffb3&chksm=fcb70848cbc0815e03947731557b16897112f37c9b67cd436013a7cf70738cada21925d37ab4&scene=21#wechat_redirect
[Link 12]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247490386&idx=2&sn=321179886eb4f6a2c6bd55aed3578097&chksm=fcb707edcbc08efb406cec43da6540df1e7c99ed88f02ebd2c313ad3549d082679a10cdf6165&scene=21#wechat_redirect
[Link 13]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247494603&idx=3&sn=1d0ca7f2eef4ace9941ce2f3ff130dbb&chksm=fcb4f774cbc37e62a1425aca8d97e593e30bd09a7b3a8f4b3d0d9ac32877979944af03030717&scene=21#wechat_redirect
[Link 14]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247495199&idx=3&sn=4e56ec540c4fa222d9e57804b554ea70&chksm=fcb4f2a0cbc37bb60193b8a653016a0d5ea933f1b9c69e39d1b06861813c78fc7c574b527454&scene=21#wechat_redirect
[Link 15]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247503912&idx=1&sn=7cd6fee3a4fd042be5ae8f54d750052b&chksm=fcb4cc97cbc345818ec45020b51297150f8b2f1f211f8d74523529f02e14a47af812a5827443&scene=21#wechat_redirect
[Link 16]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247508471&idx=1&sn=9d6a781d8015ca651f231543977b99f0&chksm=fcb4bd48cbc3345e5169a622d44d6cb1ac4ff9a31970b4cf7a132e7d766f5ae90164a1d2b662&scene=21#wechat_redirect
[Link 17]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247512691&idx=1&sn=5a9a03b656da97558fd537df2161a07e&chksm=fcb4aecccbc327da843ff574076da8701708dbd39a957ad41d6b2b9d4ac2d6e64584af2c7426&scene=21#wechat_redirect
[Link 18]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247511199&idx=1&sn=dcc96018356c7aa1f07f594ea2db53ab&chksm=fcb4b020cbc3393667267919d45479df44e03b8fccaf5ea1ecffadc4fd664e8505dc0db723ec&scene=21#wechat_redirect
[Link 19]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247510579&idx=1&sn=8b7f6facff81a9e55f653bad1c6b016f&chksm=fcb4b68ccbc33f9a2ffcfe67d87fafdd025566c76d9ad4caf9951d9ac084dae803f52ab70048&scene=21#wechat_redirect
[Link 20]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247510383&idx=1&sn=f659ab4c9b0973276be95c9bab89d5b1&chksm=fcb4b5d0cbc33cc6a9771ce2d6cd88e57b3def32c9d0e3ca8b09e413855a71f422794971fe4f&scene=21#wechat_redirect
[Link 21]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247510244&idx=1&sn=5714c36390b4a7307252c4d02cfaecf0&chksm=fcb4b45bcbc33d4de651e9b095c49a760eb4b909a1a7c0553343bd2bb3bee93b34285bbfacd3&scene=21#wechat_redirect
[Top2_2021_ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247505138&idx=1&sn=ef6102d36f7aa7191982c6b6eddb9872&chksm=fcb4c84dcbc3415be45e48c522029fb90f1c395278708f38c6d9b68d2aadaa0afff68b3ff607&scene=21#wechat_redirect
[Link 22]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247498233&idx=2&sn=7663c29445a7f07f839424656ef28cb5&chksm=fcb4e546cbc36c50b7711c59b77d01048cc81302e4a4a9f4fdbf83715360b14c5110f873b485&scene=21#wechat_redirect
[Link 23]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247497362&idx=2&sn=5b3436738550a7e4d62c2bb98befe3df&chksm=fcb4ea2dcbc3633b228e5712ecfc9d302ff56b7d96e83232f1c7ffb0a8bd2ffc908b5003b9d9&scene=21#wechat_redirect
[_ _ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247498631&idx=2&sn=7a91e5850ec4885a4d7028b925e6f2d2&chksm=fcb4e738cbc36e2e8ed92fcc6acbf5024ee73d4f5b2e05bcd2c31333fad86d6d4638b06fef6f&scene=21#wechat_redirect
[Link 24]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247489104&idx=1&sn=189ff572dc14ee137e83e7e4e7a4cc9f&chksm=fcb70aefcbc083f9b351b1b29c56d70cc1748181583490c3769b77a495e74747b8f7491ee39a&scene=21#wechat_redirect
[Link 25]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247487741&idx=2&sn=40bbb83c4ecda43fcc11fc3eb3a7130f&chksm=fcb70c42cbc085545c4ccfd51563cc59e38134147803c0c59e0aa3827385f060ccc75b3df888&scene=21#wechat_redirect
[PyTorch_Tensor]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247504061&idx=1&sn=207d2bd4895e0b4e303c3d9d63511c72&chksm=fcb4cc02cbc3451449159333c651688e7d71c7a744a4e4e0d914fb14adb5ad4f7e122eff7e18&scene=21#wechat_redirect
[CNN_ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247503173&idx=1&sn=e92e9adb903e7a87d0b64ceff982020c&chksm=fcb4d1facbc358ec09986cd8b32ce222c3394c46e2f0bd60c282d63ffeea897e7211984e6fca&scene=21#wechat_redirect
[Link 26]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247502893&idx=1&sn=e5720b57b0804dca182552ba9d6c1eb3&chksm=fcb4d092cbc359841d6f5a9e3b838e3894d34b35d74a1627fa9bf37e27c019ca9d9af84c362d&scene=21#wechat_redirect
[Transformer _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247502134&idx=1&sn=ed4a262db0be219feb017c08f2185779&chksm=fcb4d589cbc35c9f0156ac454dccb82a501f4b8129adbbcf94a3ca83d3af129f4ebadf173e75&scene=21#wechat_redirect
[Scaled-YOLOv4_ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247500056&idx=2&sn=95a04371dc0b49e8c862b03a88edbb8c&chksm=fcb4dda7cbc354b14c4cd4f63eadf02680112ee21015fe372cf3055f9b577b7dd989843eda1f&scene=21#wechat_redirect
[PyTorch_ _]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247497349&idx=3&sn=b4b1ea0af5d42c817bc776cab0a8ac28&chksm=fcb4ea3acbc3632cd61c21d40df0c8cd9e84603fcbe551ceea721e4d8c43959a3c74e929a0fe&scene=21#wechat_redirect
[PyTorch_ _ 1]: http://mp.weixin.qq.com/s?__biz=MzU2NTc5MjIwOA==&mid=2247497036&idx=2&sn=552069978cf317b9581b4c9a62a003e2&chksm=fcb4e9f3cbc360e51de7a7ceb3a6adde123e488d98a3baf4204270311ef609fa5510682ff29b&scene=21#wechat_redirect
[Numpy]: https://blog.csdn.net/dafeidouzi/article/details/100520906
[10]: https://blog.csdn.net/lvsehaiyang1993/article/details/80342473
[CVPR 2020]: https://blog.csdn.net/u013685264/article/details/104984809
[Link 27]: https://blog.csdn.net/fengdu78/article/details/107308594
[YOLOv4]: https://blog.csdn.net/fengdu78/article/details/106450074
[a1511984fdca8d782e354efee9e17b58.gif]: /images/20220828/b4a4e13dd95b494b8972ad70827f6b52.png
[cbe6d2b6e3dcd63935c71ab7da789d79.gif]: /images/20220828/e093286ba41f4d0c8eb3988cf2ade0e2.png
[7c5d4d183d6b41ad202955f2aff82af9.gif]: /images/20220828/560971effad047e8aa706d7bddca4f3a.png
[695156925f204a3a6ef126d0fee8caea.gif]: /images/20220828/df9491deaf90430db51ccf58ccb357ae.png