python 如何在decorator中使用多进程

niwlg2el 于 2023-05-16 发布在 Python

关注(0)|答案(1)|浏览(177)

我想创建一个装饰器，使函数接受单个参数，以并行处理可迭代参数。
下面是示例代码：

import functools
import time

from multiprocessing import Pool

def parallel(func):

    def wrapper(iterable):
        with Pool() as pool:
            result = pool.map(func, iterable)

        return result

    return wrapper

@parallel
def test(i):
    time.sleep(1)
    print(f"{i}: {i * i}")

def main():
    test(range(10))

if __name__ == "__main__":
    main()

但我有

Traceback (most recent call last):
  File "/home/user/projects/amdb/s2.py", line 29, in <module>
    main()
  File "/home/user/projects/amdb/s2.py", line 25, in main
    test(range(10))
  File "/home/user/projects/amdb/s2.py", line 10, in wrapper
    result = pool.map(func, iterable)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function test at 0x7fef63309120>: it's not the same object as __main__.test

我知道我可以通过不使用装饰器来解决这个问题，比如test_multi = parallel(test)，但问题是如何在装饰器中使用它。

python

来源：https://stackoverflow.com/questions/76252232/how-to-use-multiprocess-in-decorator

1条答案

按热度按时间

4ioopgfo1#

这是因为multiprocessing.Pool需要能够序列化函数和参数以便在进程之间传递，而decorator @parallel使得无法正确序列化测试函数。可以使用pathos库，它是multiprocessing的扩展，可以序列化更多类型，包括函数。请在使用前安装它。pip install pathos
验证码：

import time

from pathos.multiprocessing import ProcessPool

def parallel(func):
    def wrapper(iterable):
        with ProcessPool() as pool:
            result = pool.map(func, iterable)

        return result

    return wrapper

@parallel
def test(i):
    time.sleep(1)
    print(f"{i}: {i * i}")

def main():
    test(range(10))

if __name__ == "__main__":
    main()

赞(0）回复(0）举报 2023-05-16

我来回答

python 如何在decorator中使用多进程

1条答案

相关问题

热门标签

最新问答