Is there a multiprocessing module for Perl?

perl fork
perl fork subroutine example
multitasking in perl
perl parallel::forkmanager
perl fork exec
perl fork system command
perl return value from forked process
perl fork loop

Is there a multiprocessing module for Perl? Something that has similar functionality to what's offered by Python's multiprocessing module.

I understand I could build similar functionality using Perl, but I'm looking for something already implemented.

I think Forks::Super comes pretty close. It has a few features for running an arbitrary subroutine (or external command) in a background process, monitoring and signalling the background process, and making interprocess communication a little less painful.

use Forks::Super;

sub do_something { my @args = @_; ... }
$process = fork { sub => \&do_something, args => [@args] };
$process->wait;


my $x = 42;
my @y = ();
my %z = ();
sub do_something_else {
    $x = 19;
    @y = qw(foo bar);
    %z = (foo => 'bar');
}
$process = fork { sub => 'do_something_else', share => [\$x, \@y, \%z ] };
$process->wait;
# $x, @y, and %z are now updated with changes made in background process


# create your own pipes to use for IPC
use IO::Handle;
pipe my $child_read, my $parent_write;
pipe my $parent_read, my $child_write;
$parent_write->autoflush(1);
$child_write->autoflush(1);
sub square {
    while (my $x = <$child_read>) {
        print {$child_write} $x ** 2, "\n";
    }
    close $child_write;
}
$process = fork { sub => 'square' };
print {$parent_write} "9\n";
my $result = <$parent_read>;    # should be "81\n";
close $parent_write;

# or use the standard I/O handles for IPC
sub square_root {
    sleep 1 && seek STDIN,0,1 while eof(STDIN); # ok, this is a workaround for an existing bug :-(
    while (my $x = <STDIN>) {
        print sqrt($x), "\n";
    }
}
$process = fork { sub => 'square_root', child_fh => 'in,out,block' };
$process->write_stdin("81\n");
$result = $process->read_stdout(); #  =>  "9\n"

Both the multiprocessing module and Forks::Super have a lot of features. Which ones are you specifically interested in?

I am the author of Forks::Super and my goal is to include any features for parallel processing that people find useful, so if there's a feature in multiprocessing that you want in Perl, let me know.

Is there a multiprocessing module for Perl?, Is there a multiprocessing module for Perl? perl fork perl fork subroutine example multitasking in perl perl fork exec perl fork system command perl parallel::  For more information on multiprocessing with fork and interprocess communication, I recommend Lincoln Stein's book, "Network Programming with Perl." And in order to do advanced multiprocessing tasks (like spawning a daemon) you'll need to understand the underlying process model in more detail than we have gone into here.

forks provides the same awesome interface as threads, but uses processes instead of threads.

use forks;  # Or: use threads;
use Thread::Queue;

my $q = Thread::Queue->new();

my @workers;
for (1..NUM_WORKERS) {
   push @workers, async {
      while (defined(my $job = $q->dequeue())) {
         ...
      }
   };
}

$q->enqueue(...);

$q->enqueue(undef) for @workers;
$_->join for @workers;

Comparing forks with Forks::Super.

Keep in mind, these are suppose to the be the cases where Forks::Super excels!

use Forks::Super;
sub do_something { my @args = @_; ... }
$process = fork { sub => \&do_something, args => [@args] };
$process->wait;

can be written as

use forks;
sub do_something { my @args = @_; ... }
$process = async { do_something(@args) };
$process->join;

---

use Forks::Super;
my $x = 42;
my @y = ();
my %z = ();
sub do_something_else {
    $x = 19;
    @y = qw(foo bar);
    %z = (foo => 'bar');
}
$process = fork { sub => 'do_something_else', share => [\$x, \@y, \%z ] };
$process->wait;

can be written as

use forks;
use forks::shared;
my $x :shared = 42;
my @y :shared = ();
my %z :shared = ();
sub do_something_else {
    $x = 19;
    @y = qw(foo bar);
    %z = (foo => 'bar');
}
$process = async { do_something_else() };
$process->join;

---

use Forks::Super;
use IO::Handle;
pipe my $child_read, my $parent_write;
pipe my $parent_read, my $child_write;
$parent_write->autoflush(1);
$child_write->autoflush(1);
sub square {
    while (my $x = <$child_read>) {
        chomp($x);
        print {$child_write} $x ** 2, "\n";
    }
    close $child_write;
}
$process = fork { sub => 'square' };
print { $parent_write } "9\n";
chomp( my $result = <$parent_read> );  # 81
close $parent_write;
$process->wait;

can be written as

use forks;
use Threads::Queue;
my $req = Threads::Queue->new();
my $resp = Threads::Queue->new();
sub square { $_[0] ** 2 }
$process = async {
    while (defined(my $x = $req->dequeue())) {
        $resp->enqueue( square($x) );
    }
};
$req->enqueue(9);
my $result = $resp->dequeue();  # 81
$resp->enqueue(undef);
$process->join;

---

use Forks::Super;
sub square_root {
    sleep 1 && seek STDIN,0,1 while eof(STDIN); # ok, this is a workaround for an existing bug :-(
    while (my $x = <STDIN>) {
        chomp($x);
        print sqrt($x), "\n";
    }
}
$process = fork { sub => 'square_root', child_fh => 'in,out,block' };
$process->write_stdin("81\n");
chomp( $result = $process->read_stdout() );  # 9
$process->close_fh('stdin');
$process->wait;

can be written as

use forks;
use Threads::Queue;
my $req = Threads::Queue->new();
my $resp = Threads::Queue->new();
$process = async {
    while (defined(my $x = $req->dequeue())) {
        $resp->enqueue( sqrt($x) );
    }
};
$req->enqueue(81);
my $result = $resp->dequeue();  # 9
$resp->enqueue(undef);
$process->join;

Multiprocessing and Multitasking, When you run an external program via system, Perl does a fork; in the child, Perl does In our example, we didn't wait for the child or trap signals; if the parent  Perl modules may also contain documentation, so they can be used by multiple programmers without each programmer needing to read all of the code. Modules are the foundation of the CPAN, which contains thousands of ready-to-use modules, many of which you will likely use on a regular basis.

What about POE: Perl Object Environment? It has support for asynchronous child processes.

Multiprocess module, I'm looking for a module for Perl with multiprocess support. Now I'm using parallel​::forkmanager but this module does not communicate with the  This module provides a non-compliant wrapper around the widely implemented MPI libraries, allowing messages to consist of arbitarily nested Perl data structures whose size is limited by available memory. This hybrid approach should allow you to quickly write programs which run anywhere which supports MPI (both Beowulf and traditional MPP machines).

You can use https://github.com/marioroy/mce-perl It is similar to python multiprocess module

Chip Multiprocessor Generator: Automatic Generation of Custom and , In that case either modify it to identify .vp and .vph files, or replace with the above SomeModule.pm Files Genesis2 generates intermediate Perl module files. Multiprocessing in Python is a package we can use with Python to spawn processes using an API that is much like the threading module. With support for both local and remote concurrency, it lets the programmer make efficient use of multiple processors on a given machine.

Network Programming with Perl, About Threads Multithreading is quite different from multiprocessing. Instead of there The Thread module was introduced in Perl 5.005. To use it, you must run​  I encountered a weird problem while using python multiprocessing library. My code is sketched below: I spawn a process for each "symbol, date" tuple. I combine the results afterwards.

Developing Web Applications with Apache, MySQL, memcached, and Perl, This makes it possible for module developers to alter Apache's basic functionality​. Many existing books available for Perl web development, CGI, and mod_perl has four execution / MPM (Multiprocessing Modules) models: El MPM Worker:​  multiprocessing.map(lambda f: f(1), functions) except that lambda functions aren't pickleable, so you'd need to use a defined function (see How to let Pool.map take a lambda function). You can actually use the builtin apply() (not the multiprocessing one) (although it's deprecated): multiprocessing.map(apply,[(f,1) for f in functions])

perlthrtut, The threads and threads::shared modules are included in the core Perl distribution. Multiprocessor kernel threads are the final step in thread support. Introduction¶. multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads.

Comments
  • Don't forget that Perl has fork and exec right out of the box.
  • Yeah that's the one I added it to the question as well.
  • @PlatinumAzure The tricky part is "The multiprocessing package offers both local and remote concurrency," I don't know if there is a portable abstraction for the remote part.
  • If I needed remote concurrency, I'd start looking at something like gearman and Gearman::XS. If local, then fork/exec or maybe Parallel::ForkManager to manage the processes.
  • In addition to fork() there are threads (assuming that your Perl has been compiled for that support). See 'perlthrtut' DOC
  • This is just what I was looking for. I got this task to add parallelism to improve running times on some Perl scripts we have. I had used Python's multiprocessing before, but nothing like that in Perl. I'll have more specific questions as I start to convert the scripts.
  • Ugh, I just checked if this module is installed in my Perl, and apparently it's not. I wish essential stuff like this was part of the standard Perl package. :(
  • I'm using Forks::Super on Windows 10 with Strawberry Perl 5.30, and it seems when I use Forks::Super::fork( 'sub' => \&myFunc ... ); it doesn't actually launch a child process. Is it possible that on Windows, Forks::Super merely emulates the normal Unix fork behavior via threads? :-\
  • Yes, to run a Perl subroutine in the background on Windows, Forks::Super will use the builtin fork call which spawns a pseudo-process for the child code to run in.
  • @mob And I suppose that means if I used Forks::Super to parallelize a bunch of work to take advantage of the CPU's multiple cores then this emulation behavior kind of defeats that purpose, since this pseudo-process is really just a thread, and Perl's threads do not really run in parallel (due to some global interpreter lock, similar to Python's GIL), if I understand correctly.
  • @jkysam, Added comparison of forks and Forks::Super.
  • Welcome to Stack Overflow. While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Answers that are little more than a link may be deleted.