Use Python multiprocessing library¶
Base class for all masters implemented with Python multiprocessing.Process.
It creates a number of workers each with an individual Process. The communication between master and each worker is via independent Pipe connection. The master assigns tasks to workers. When all tasks are done, it stops all workers and terminate all processes.
If there are more tasks than workers, then tasks will be splitted into chunks. If there are less tasks than workers, then we reduce the number of workers to the number of tasks.
Assign a given list of tasks to the workers and return the received results.
Parameters: tasks (list) – a list of tasks Returns: results – received results Return type: object
Defines everything required after finishing all the works, e.g. stop all workers, clean up.
Returns a list of tasks.
Returns: tasks – a list of tasks Return type: list
Base class for all workers implemented with Python multiprocessing.Process.
It communicates with master via a Pipe connection. The worker is stand-by infinitely waiting for task from master, working and sending back result. When it receives a
closecommand, it breaks the infinite loop and close the connection.
Work on the given task and return the result.
- task_id (int) – the task ID.
- task (object) – a given task.
result – working result.