Comments on: Parallelize by process http://nutrun.com/weblog/parallelize-by-process/ nutrun Mon, 28 Jun 2010 07:35:37 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Ochronus http://nutrun.com/weblog/parallelize-by-process/comment-page-1/#comment-1221 Ochronus Mon, 27 Oct 2008 13:29:39 +0000 http://nutrun.com/?p=189#comment-1221 Nice article (as usual), I've created something similar (actually a ParallelRunner class), it has an init and run phase, and two overridable methods (success and failure) to handle the outcome of each separate process. Basically init fills an internal command array, and run then runs them in the background with a given concurrency. We use it from image downloading to feed processing. It's a lot more easier, since it separates the "complicated" concurrency handling routine from the worker code, and also there are no multithreading issues, plus every process is responsible for what it's doing 100%. Nice article (as usual), I’ve created something similar (actually a ParallelRunner class), it has an init and run phase, and two overridable methods (success and failure) to handle the outcome of each separate process. Basically init fills an internal command array, and run then runs them in the background with a given concurrency. We use it from image downloading to feed processing. It’s a lot more easier, since it separates the “complicated” concurrency handling routine from the worker code, and also there are no multithreading issues, plus every process is responsible for what it’s doing 100%.

]]> By: George Malamidis http://nutrun.com/weblog/parallelize-by-process/comment-page-1/#comment-1219 George Malamidis Sun, 26 Oct 2008 21:02:12 +0000 http://nutrun.com/?p=189#comment-1219 Thanks for bringing this up, I've amended the example to reflect your observation. Thanks for bringing this up, I’ve amended the example to reflect your observation.

]]> By: Shadowfiend http://nutrun.com/weblog/parallelize-by-process/comment-page-1/#comment-1216 Shadowfiend Sun, 26 Oct 2008 16:58:17 +0000 http://nutrun.com/?p=189#comment-1216 Seems like your final example will have an issue with spawning more or less workers than the count necessarily allows for if access to the @worker_count variable isn't protected by a mutex. This would happen if two process completed at approximately the same time, causing their spawning threads to read the worker count simultaneously, then both increment it and save it back. This way, the worker count variable would only change by 1 when two processes actually terminated. Also worth mentioning in these contexts is the MapReduce algorithm (http://en.wikipedia.org/wiki/MapReduce) and Ruby implementations like Skynet (http://skynet.rubyforge.org/). Seems like your final example will have an issue with spawning more or less workers than the count necessarily allows for if access to the @worker_count variable isn’t protected by a mutex. This would happen if two process completed at approximately the same time, causing their spawning threads to read the worker count simultaneously, then both increment it and save it back. This way, the worker count variable would only change by 1 when two processes actually terminated.

Also worth mentioning in these contexts is the MapReduce algorithm (http://en.wikipedia.org/wiki/MapReduce) and Ruby implementations like Skynet (http://skynet.rubyforge.org/).

]]>