Passing a static class method to multiprocessing.Process in Python 3.4

Question:

I'm trying to run a static class method via multiprocessing.Process .

import multiprocessing as mp

class Test:
    @staticmethod
    def method():
        pass

if __name__=="__main__":       
    proc = mp.Process(target=Test.method)
    proc.start()

But I'm facing the fact that it works in python 3.6, but does not work in 3.4 (and the project requires python support from 3.4). Looking through the documentation, I found that only methods defined in the top level of the module can be passed . But then why does it work in python 3.6? After looking at the changelogs, I did not find a change in this mechanic. Accordingly, questions arose:

  1. Is it possible to somehow wrap an arbitrary function or method so that it can be passed to Process .

For a single method, you can make a wrapper function at the module level:

import multiprocessing as mp

class Test:
    @staticmethod
    def method():
        pass

def method_wrapper(): # <======= This
    return Test.method()

if __name__=="__main__":       
    proc = mp.Process(target=method_wrapper)
    proc.start()

but this will not work when there are many such functions. Is it possible to make some kind of function that will take the method I need and return something that can be passed to Process

  1. Does the behavior of python 3.6 match what's in the documentation (or is it a bug/feature)? That is, can I expect that this behavior will not change in the future?

PS

Why do I need it? Globally, the task is to limit the execution time of functions. That is, there is a script that performs different actions, and some of them can be very long, and you need to kill them if they take longer than a certain time, and do something with it (either wait and try again, or start rolling back) . That is, what I'm trying to do: run a function/method in a separate process, wait for it in the main for a while and kill it if necessary.

Threads don't fit, I can't kill them. I need to ensure that if the time expires the action is no longer performed before working on.

In general, if there are ideas on how to do it differently, I will be glad to ideas.

Answer:

If I understand correctly, you get an error trace like this (tested on Python 3.4.4, Windows 7 64-bit):

Traceback (most recent call last):
  File "multiproc.py", line 17, in <module>
    proc.start()
  File "c:\Python34\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "c:\Python34\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "c:\Python34\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "c:\Python34\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
    reduction.dump(process_obj, to_child)
  File "c:\Python34\lib\multiprocessing\reduction.py", line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function Test.method at 0x0000000002A84730>
: attribute lookup method on __main__ failed

...
  File "<string>", line 1, in <module>
  File "c:\Python34\lib\multiprocessing\spawn.py", line 100, in spawn_main
    new_handle = steal_handle(parent_pid, pipe_handle)
  File "c:\Python34\lib\multiprocessing\reduction.py", line 81, in steal_handle
    _winapi.PROCESS_DUP_HANDLE, False, source_pid)
OSError: [WinError 87] Параметр задан неверно

Note the line Can't pickle <function Test.method at 0x0000000002A84730> .

Indeed, prior to Python 3.5, pickle did not support wrapping bound methods. Since version 3.5 it supports .

Total

  1. If there is a need to use version 3.4, then you can use one of the tips: Can't pickle when using multiprocessing Pool.map() or Pickling a staticmethod in Python (however, you will need to supplement the code with wrappers).

  2. Starting from version 3.5 everything works and will work.

Options for improvements for Python 3.4

  1. Wrapper

As suggested by @jfs in the comment above:

def wrapper(klass, method):
    getattr(klass, method)()


if __name__ == "__main__":   
    proc = mp.Process(target=wrapper, args=(Test, "method"))
    proc.start()
  1. Use metaclass

According to the answer from the main Stackoverflow: http://stackoverflow.com/a/1914798/711380

class _PickleableStaticMethod(object):
    def __init__(self, fn, cls=None):
        self.cls = cls
        self.fn = fn
    def __call__(self, *args, **kwargs):
        return self.fn(*args, **kwargs)
    def __get__(self, obj, cls):
        return _PickleableStaticMethod(self.fn, cls)
    def __getstate__(self):
        return (self.cls, self.fn.__name__)
    def __setstate__(self, state):
        self.cls, name = state
        self.fn = getattr(self.cls, name).fn


class pickleable_staticmethods(type):
    def __new__(cls, name, bases, dct):
        new_cls = type.__new__(cls, name, bases, dct)
        dct = new_cls.__dict__
        for name in dct.keys():
            value = new_cls.__dict__[name]
            if isinstance(value, staticmethod):
                setattr(
                        new_cls,
                        name,
                        _PickleableStaticMethod(value.__get__(None, new_cls),
                                               new_cls))
        return new_cls



class Test(metaclass=pickleable_staticmethods):
    @staticmethod
    def method():
        while True:
            print(time.ctime())
            time.sleep(2)


if __name__ == "__main__":   
    proc = mp.Process(target=Test.method)
    proc.start()
Post Views: 15
Scroll to Top