[Maya-Python] Multiprocessing Returns New Maya Instance

[Ruchit Bhatt]

Hi,
I tried multiprocessing module in maya python with basic example and every
time it returns new maya instance plus

object of multiprocessing.Queue() returns nothing on object.get()
is python interpreter is not a separate process in Maya ?
please share efficient example on how to deal with multiprocessing in Maya.
Thank you

[Marcus Ottosson]

The Python interpreter and Maya run in the same process, so this is
expected.

If your multiprocessing doesn't require access to your current session,
then you may consider instead launching mayapy explicitly (either from or
outside of Maya), and from there use the multiprocessing module. That would
create additional instances of mayapy instead which may be what you want.

Post by Ruchit Bhatt
Hi,
I tried multiprocessing module in maya python with basic example and every
time it returns new maya instance plus
object of multiprocessing.Queue() returns nothing on object.get()
is python interpreter is not a separate process in Maya ?
please share efficient example on how to deal with multiprocessing in Maya.
Thank you
--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit https://groups.google.com/d/
msgid/python_inside_maya/ca05daed-3c24-43de-829c-
9ca8aa2f7725%40googlegroups.com
<https://groups.google.com/d/msgid/python_inside_maya/ca05daed-3c24-43de-829c-9ca8aa2f7725%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Justin Israel]

Post by Marcus Ottosson
The Python interpreter and Maya run in the same process, so this is
expected.
If your multiprocessing doesn't require access to your current session,
then you may consider instead launching mayapy explicitly (either from or
outside of Maya), and from there use the multiprocessing module. That
would create additional instances of mayapy instead which may be what you
want.

Post by Ruchit Bhatt
Hi,
I tried multiprocessing module in maya python with basic example and
every time it returns new maya instance plus
object of multiprocessing.Queue() returns nothing on object.get()
is python interpreter is not a separate process in Maya ?
please share efficient example on how to deal with multiprocessing in Maya.
Thank you

You should be able to follow this example:
https://pymotw.com/2/multiprocessing/communication.html#passing-messages-to-processes

The queue has to be passed to the child process as an arg, because it will
be a completely separate process with its own memory.

One other thing to note... be careful when choosing to use multiprocessing
within Maya. Starting a Process() results in a fork of the current Maya
process, which means if you have a huge scene open, you are duplicating
that memory in your child process. It can also result in deadlocks because
of the heavily threaded environment in Maya, and locks that were acquired
may not be released in the new child process (at least this is true on
osx/linux).

If you are set on using multiprocessing, try to create a pool very early
before any memory is allocated in your Maya session. Or better yet, go the
route Marcus suggested and launch a mayapy in a subprocess.

Justin

Post by Marcus Ottosson

Post by Ruchit Bhatt
--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/ca05daed-3c24-43de-829c-9ca8aa2f7725%40googlegroups.com
<https://groups.google.com/d/msgid/python_inside_maya/ca05daed-3c24-43de-829c-9ca8aa2f7725%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCwVF05XD4aYq9Bmr94-jpHmPB526AXLxwb%3D5HwXRgvnQ%40mail.gmail.com
<https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCwVF05XD4aYq9Bmr94-jpHmPB526AXLxwb%3D5HwXRgvnQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Ruchit Bhatt]

This way ?

import platform
import multiprocessing
multiprocessing.set_executable("C:/Program
Files/Autodesk/Maya201x/bin/mayapy.exe")
multiprocessing.set_executable('/usr/Autodesk/maya201x/bin/mayapy')

[Justin Israel]

Post by Ruchit Bhatt
This way ?

import platform
import multiprocessing
multiprocessing.set_executable("C:/Program
Files/Autodesk/Maya201x/bin/mayapy.exe")
multiprocessing.set_executable('/usr/Autodesk/maya201x/bin/mayapy')

Well no. The suggestion was to use the subprocess module to avoid forking

the potentially large Maya process and possibly hitting deadlocks.

I haven't had to use set_executable so I am not sure if that is necessary
or not.
--

Post by Ruchit Bhatt
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/cacf4c80-cacc-4593-857a-509782ba7a87%40googlegroups.com
<https://groups.google.com/d/msgid/python_inside_maya/cacf4c80-cacc-4593-857a-509782ba7a87%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Ruchit Bhatt]

I tried your example & its working fine. Thanks

*In script Editor*
import subprocess
import cPickle

mayapy_path = r'C:/Program Files/Autodesk/Maya201x/bin/mayapy.exe'
script_path = r'E:/multi_test.py'

proc_obj = subprocess.Popen(mayapy_path + ' ' + script_path + ' -po', stdin=
subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
result = cPickle.load(proc_obj.stdout)
print result

*multi_test.py*
import multiprocessing
from time import time
import cPickle
import sys

## Simple func to eat up cpu power.
def whileFunc(z):
while z < 100000:
z += 1
return z

if __name__ == "__main__":
## Get current time
currtime = time()

## How often to run (just a test value)
N = 10000
## Just a list with 1s
myList = [1]*N

nrOfProcessors = multiprocessing.cpu_count()

## Set our pool of processors
po = multiprocessing.Pool(nrOfProcessors)

## create the threads
res = po.map_async(whileFunc, myList)

## If we pass a -po flag, pickle the output and write it out
if '-po' in sys.argv[1:]:
results = len(res.get())
cPickle.dump(results, sys.stdout, -1)
sys.stdout.flush()
sys.exit(0)

print 'This value below should be a 1000:'
print len(res.get())
print 'time elapsed:', time() - currtime

[Justin Israel]

Post by Ruchit Bhatt
I tried your example & its working fine. Thanks
*In script Editor*
import subprocess
import cPickle
mayapy_path = r'C:/Program Files/Autodesk/Maya201x/bin/mayapy.exe'
script_path = r'E:/multi_test.py'
proc_obj = subprocess.Popen(mayapy_path + ' ' + script_path + ' -po',
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
result = cPickle.load(proc_obj.stdout)
print result
*multi_test.py*
import multiprocessing
from time import time
import cPickle
import sys
## Simple func to eat up cpu power.
z += 1
return z
## Get current time
currtime = time()
## How often to run (just a test value)
N = 10000
## Just a list with 1s
myList = [1]*N
nrOfProcessors = multiprocessing.cpu_count()
## Set our pool of processors
po = multiprocessing.Pool(nrOfProcessors)
## create the threads
res = po.map_async(whileFunc, myList)
## If we pass a -po flag, pickle the output and write it out
results = len(res.get())
cPickle.dump(results, sys.stdout, -1)
sys.stdout.flush()
sys.exit(0)
print 'This value below should be a 1000:'
print len(res.get())
print 'time elapsed:', time() - currtime

Awesome. Glad that it worked with subprocess. One more suggestion. You
might want to try calling it like this :


proc_obj = subprocess.Popen([mayapy_path, script_path, ' -po'] ,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc_obj.communicate()
result = cPickle.load(out)

If you don't need to send stdin to your process, no need to create a pipe.
And you should use communicate to avoid a deadlock of your child process
produces lots of stdout and stderr.

Post by Ruchit Bhatt
--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/2f4122d0-ff64-4ba9-8bfa-71e7bad7f0f7%40googlegroups.com
<https://groups.google.com/d/msgid/python_inside_maya/2f4122d0-ff64-4ba9-8bfa-71e7bad7f0f7%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

[Ruchit Bhatt]

Post by Justin Israel
If you don't need to send stdin to your process, no need to create a pipe.
And you should use communicate to avoid a deadlock of your child process
produces lots of stdout and stderr.
Thanks for this tips, i was not aware of it.