lagom.multiprocessing

Use Python multiprocessing library

class lagom.multiprocessing.ProcessMaster(worker_class, num_worker)

Base class for all masters implemented with Python multiprocessing.Process.

It creates a number of workers each with an individual Process. The communication between master and each worker is via independent Pipe connection. The master assigns tasks to workers. When all tasks are done, it stops all workers and terminate all processes.

Note

If there are more tasks than workers, then tasks will be splitted into chunks. If there are less tasks than workers, then we reduce the number of workers to the number of tasks.

assign_tasks(tasks)

Assign a given list of tasks to the workers and return the received results.

Parameters:tasks (list) – a list of tasks Returns:results – received results Return type:object

close()

Defines everything required after finishing all the works, e.g. stop all workers, clean up.

make_tasks()

Returns a list of tasks.

Returns:tasks – a list of tasks Return type:list

class lagom.multiprocessing.ProcessWorker(master_conn, worker_conn)

Base class for all workers implemented with Python multiprocessing.Process.

It communicates with master via a Pipe connection. The worker is stand-by infinitely waiting for task from master, working and sending back result. When it receives a close command, it breaks the infinite loop and close the connection.

work(task_id, task)

Work on the given task and return the result.

Parameters:

• task_id (int) – the task ID.
• task (object) – a given task.

Returns:

result – working result.

Return type:

object