python instance method_Python Can't pickle type 'instancemethod' 问题

比眉伴天荒 2023-01-10 03:40 86阅读 0赞

最近给Simiki从单进程改为多进程时, 遇到这个报错.

给一个正常的简化的版本:

\#!/usr/bin/env python

\# -\*- coding: utf-8 -\*-

import multiprocessing

def func(x):

print(x \* x)

class Klass(object):

def \_\_init\_\_(self):

print "Constructor ... %s" % multiprocessing.current\_process().name

def \_\_del\_\_(self):

print "... Destructor %s" % multiprocessing.current\_process().name

def func(self, x):

print(x \* x)

def run(self):

pool = multiprocessing.Pool(processes=3)

for num in range(8):

pool.apply\_async(func, args=(num,))

pool.close()

pool.join()

if \_\_name\_\_ == '\_\_main\_\_':

\_kls = Klass()

\_kls.run()

这里加上构造函数和析构函数, 是因为后面在这里会有问题.

另外, 这里有个比较奇怪的现象, 原先实例类时, 变量名是kls, 但是析构出错, 改为als没有问题, 试了很多变量名, 有的有问题, 有的没问题, 还不清楚原因

Constructor ... MainProcess

Exception AttributeError: "'NoneType' object has no attribute 'current\_process'" in > ignored

原因已经找到, 参考\_\_del\_\_文档:

Starting with version 1.5, Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the \_\_del\_\_() method is called.

对于普通变量, Python无法保证\_\_del\_\_和对象销毁的顺序, 如果变量名以下划线开头, 可以保证此变量在其它导入模块之前调用.

具体可以看看我在StackOverflow上的提问

正常执行结果:

Constructor ... MainProcess

... Destructor MainProcess

如果把func移到类中, 改为method, 则问题来了:

\#!/usr/bin/env python

\# -\*- coding: utf-8 -\*-

import multiprocessing

class Klass(object):

def \_\_init\_\_(self):

print "Constructor ... %s" % multiprocessing.current\_process().name

def \_\_del\_\_(self):

print "... Destructor %s" % multiprocessing.current\_process().name

def func(self, x):

print(x \* x)

def run(self):

pool = multiprocessing.Pool(processes=3)

for num in range(8):

pool.apply\_async(self.func, args=(num,))

pool.close()

pool.join()

if \_\_name\_\_ == '\_\_main\_\_':

\_kls = Klass()

\_kls.run()

执行报错:

Constructor ... MainProcess

Exception in thread Thread-2:

Traceback (most recent call last):

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in \_\_bootstrap\_inner

self.run()

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 761, in run

self.\_\_target(\*self.\_\_args, \*\*self.\_\_kwargs)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 342, in \_handle\_tasks

put(task)

PicklingError: Can't pickle : attribute lookup \_\_builtin\_\_.instancemethod failed

原因是multiprocessing会对调用的函数做序列化, 然后method不可被序列化. 具体参考python2 - pickle的文档, 里面列出了可被序列化的对象:

None, True, and False

integers, long integers, floating point numbers, complex numbers

normal and Unicode strings

tuples, lists, sets, and dictionaries containing only picklable objects

functions defined at the top level of a module

built-in functions defined at the top level of a module

classes that are defined at the top level of a module

instances of such classes whose \_\_dict\_\_ or the result of calling \_\_getstate\_\_() is picklable (see section The pickle protocol for details).

关于解决思路, 众说纷坛, 有利有弊.

最简单的思路? 感谢这个兄弟的思路, 加一个全局的代理函数, 他也注意到了后面的几个方法会出现析构多次, 但是这个其实也会这样:

\#!/usr/bin/env python

\# -\*- coding: utf-8 -\*-

import multiprocessing

def proxy(cls\_instance, i):

return cls\_instance.func(i)

class Klass(object):

def \_\_init\_\_(self):

print "Constructor ... %s" % multiprocessing.current\_process().name

def \_\_del\_\_(self):

print "... Destructor %s" % multiprocessing.current\_process().name

def func(self, x):

print(x \* x)

def run(self):

pool = multiprocessing.Pool(processes=3)

for num in range(8):

\#pool.apply\_async(self.func, args=(num,))

pool.apply\_async(proxy, args=(self, num,))

pool.close()

pool.join()

if \_\_name\_\_ == '\_\_main\_\_':

\_kls = Klass()

\_kls.run()

执行代码:

Constructor ... MainProcess

... Destructor PoolWorker-1

... Destructor PoolWorker-2

... Destructor PoolWorker-1

... Destructor PoolWorker-3

... Destructor PoolWorker-2

... Destructor PoolWorker-1

... Destructor PoolWorker-3

... Destructor PoolWorker-2

... Destructor MainProcess

注意到这里线程池中的worker析构了8次, 每个worker进程析构了2-3次, 主进程析构了1次.

还有一个介绍的比较多的就是用copy\_reg将MethodType注册为可序列化来解决:

参考这个回答的模板:

\#!/usr/bin/env python

\# -\*- coding: utf-8 -\*-

import copy\_reg

import types

def \_pickle\_method(m):

if m.im\_self is None:

return getattr, (m.im\_class, m.im\_func.func\_name)

else:

return getattr, (m.im\_self, m.im\_func.func\_name)

copy\_reg.pickle(types.MethodType, \_pickle\_method)

或者这个回答的模板:

\#!/usr/bin/env python

\# -\*- coding: utf-8 -\*-

from copy\_reg import pickle

from types import MethodType

def \_pickle\_method(method):

func\_name = method.im\_func.\_\_name\_\_

obj = method.im\_self

cls = method.im\_class

return \_unpickle\_method, (func\_name, obj, cls)

def \_unpickle\_method(func\_name, obj, cls):

for cls in cls.mro():

try:

func = cls.\_\_dict\_\_\[func\_name\]

except KeyError:

pass

else:

break

return func.\_\_get\_\_(obj, cls)

pickle(MethodType, \_pickle\_method, \_unpickle\_method)

这里用到了mro()

im\_self is the class instance object;

im\_func is the function object;

im\_class is the class of im\_self for bound methods or the class that asked for the method for unbound methods;

具体见文档

\#!/usr/bin/env python

\# -\*- coding: utf-8 -\*-

class Klass(object):

def func(self):

pass

def main(self):

print 'self.func.im\_self: %s' % self.func.im\_self

print 'self.func.im\_class: %s' % self.func.im\_class

print 'self.func.im\_func: %s' % self.func.im\_func

\_kls = Klass()

\_kls.main()

print '---'

print '\_kls.func.im\_self: %s' % \_kls.func.im\_self

print '\_kls.func.im\_class: %s' % \_kls.func.im\_class

print '\_kls.func.im\_func: %s' % \_kls.func.im\_func

print '---'

print 'Klass.func.im\_self: %s' % Klass.func.im\_self

print 'Klass.func.im\_class: %s' % Klass.func.im\_class

print 'Klass.func.im\_func: %s' % Klass.func.im\_func

执行结果:

self.func.im\_self: <\_\_main\_\_.klass object at>

self.func.im\_class:

self.func.im\_func:

\---

\_kls.func.im\_self: <\_\_main\_\_.klass object at>

\_kls.func.im\_class:

\_kls.func.im\_func:

\---

Klass.func.im\_self: None

Klass.func.im\_class:

Klass.func.im\_func:

接着回来, 上面的方法, 即:

\#!/usr/bin/env python

\# -\*- coding: utf-8 -\*-

import multiprocessing

import copy\_reg

import types

def \_pickle\_method(m):

if m.im\_self is None:

return getattr, (m.im\_class, m.im\_func.func\_name)

else:

return getattr, (m.im\_self, m.im\_func.func\_name)

copy\_reg.pickle(types.MethodType, \_pickle\_method)

class Klass(object):

def \_\_init\_\_(self):

print "Constructor ... %s" % multiprocessing.current\_process().name

def \_\_del\_\_(self):

print "... Destructor %s" % multiprocessing.current\_process().name

def func(self, x):

print(x \* x)

def run(self):

pool = multiprocessing.Pool(processes=3)

for num in range(8):

pool.apply\_async(self.func, args=(num,))

pool.close()

pool.join()

if \_\_name\_\_ == '\_\_main\_\_':

\_kls = Klass()

\_kls.run()

执行代码:

Constructor ... MainProcess

... Destructor PoolWorker-1

... Destructor PoolWorker-2

... Destructor PoolWorker-1

... Destructor PoolWorker-3

... Destructor PoolWorker-1

... Destructor PoolWorker-2

... Destructor PoolWorker-1

... Destructor PoolWorker-3

... Destructor MainProcess

同上.

另一个方法是, 参考这个回答, 定义\_\_call\_\_. 这样就可以将类本身, 如这里的self传给apply\_async, 因为顶层定义的类是可被序列化的:

\#!/usr/bin/env python

\# -\*- coding: utf-8 -\*-

import multiprocessing

class Klass(object):

def \_\_init\_\_(self):

print "Constructor ... %s" % multiprocessing.current\_process().name

def \_\_del\_\_(self):

print "... Destructor %s" % multiprocessing.current\_process().name

def \_\_call\_\_(self, x):

return self.func(x)

def func(self, x):

print(x \* x)

def run(self):

pool = multiprocessing.Pool(processes=3)

for num in range(8):

\#pool.apply\_async(self.func, args=(num,))

pool.apply\_async(self, args=(num,))

pool.close()

pool.join()

if \_\_name\_\_ == '\_\_main\_\_':

\_kls = Klass()

\_kls.run()

执行代码:

Constructor ... MainProcess

... Destructor PoolWorker-1

... Destructor PoolWorker-2

... Destructor PoolWorker-3

... Destructor PoolWorker-1

... Destructor PoolWorker-2

... Destructor PoolWorker-3

... Destructor PoolWorker-1

... Destructor PoolWorker-2

... Destructor MainProcess

这个回答的方法其实和之前设置的代理函数一样, 都是把method提升到global:

\#!/usr/bin/env python

\# -\*- coding: utf-8 -\*-

import multiprocessing

def call\_it(instance, name, args=(), kwargs=None):

"""indirect caller for instance methods and multiprocessing"""

if kwargs is None:

kwargs = \{\}

return getattr(instance, name)(\*args, \*\*kwargs)

class Klass(object):

def \_\_init\_\_(self):

print "Constructor ... %s" % multiprocessing.current\_process().name

def \_\_del\_\_(self):

print "... Destructor %s" % multiprocessing.current\_process().name

def func(self, x):

print(x \* x)

def run(self):

pool = multiprocessing.Pool(processes=3)

for num in range(8):

\#pool.apply\_async(self.func, args=(num,))

pool.apply\_async(call\_it, args=(self, 'func', (num,)))

pool.close()

pool.join()

if \_\_name\_\_ == '\_\_main\_\_':

\_kls = Klass()

\_kls.run()

执行代码:

Constructor ... MainProcess

... Destructor PoolWorker-1

... Destructor PoolWorker-3

... Destructor PoolWorker-2

... Destructor PoolWorker-1

... Destructor PoolWorker-3

... Destructor PoolWorker-1

... Destructor PoolWorker-2

... Destructor MainProcess

关于析构8次的问题，看了copy\_reg, pickle, multiprocessing的代码, 还是没什么头绪. 坑爹!

这块考虑, 最还还是把多进程调用的函数抽离出来放在全局是最安全的; 毕竟其它的方案又丑陋, 并且最主要的是各种黑魔法, 完全不清楚内部细节.

另外, Python3.3+ 解决了这个问题, 所以在Python3下是可运行的, 这里有个讨论

python instance method_Python Can't pickle type 'instancemethod' 问题

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关 Python常用函数：pickle【序列化：pickle.dump】【反序列化：pickle.load】

相关 python instance method_Python Can't pickle type 'instancemethod' 问题

相关 Python2 PicklingError: Can‘t pickle ＜type ‘instancemethod‘＞: attribute lookup builtin.instanceme

相关 python pickle protocol_Python，如何处理“ValueError:unsupported pickle protocol:4”错误？

相关 Python的pickle使用

相关 Python——Pickle库

相关 python之pickle

相关 58-Python的pickle

相关 python Pickle标准模块

相关 python pickle模块