Multiprocessing in Python is really easy! You can spawn processes by using the Pool() class.
1 2 3 4 5 6 7 8 9 10 11 |
from multiprocessing import Pool def add_one(to_what): """A trivial function to increment an integer""" return to_what + 1 pool = Pool(processes=4) #spawn 4 processes numbers = [1,2,3,4] #this is our test data result = pool.map(add_one,numbers) #send the function and 1 number to each pool.close() #kill the processes print result #will return [2,3,4,5] |
Here, we spawn 4 processes, and use the map() function to send a number to each of them.
This is a trivial example, but gets much more powerful when each process does something like making a remote connection:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
from multiprocessing import Pool from pymongo import Connection def how_many(server_number): """Returns how many documents in the collection""" c = Connection("192.168.0." + server_number) #connects to remote DB return c.MyDB.MyCollection.count() pool = Pool(processes=4) servers = [1,2,3,4] result = pool.map(how_many,servers) pool.close() print "You have {0} docs across all MongoDB servers!".format(sum(result)) |