Ryfujk

I have a few test scripts, each of which runs a test PHP app. Each script runs forever.

So, cat.sh, dog.sh, and foo.sh each run a PHP script. Each shell script runs the PHP app in a loop, so it runs forever, sleeping after each run.

I'm trying to figure out how to run the scripts in parallel at the same time. See the output of the PHP apps in the stdout/term window.

I thought, doing something like the following might work:

foo.sh > &2

dog.sh > &2

cat.sh > &2

In a shell script it would be sufficient, but it's not working.

foo.sh, runs foo.php once, and it runs correctly

dog.sh, runs dog.php in a never ending loop. it runs as expected

cat.sh, runs cat.php in a never ending loop

 *** this never runs!!!

It appears that the shell script never gets to run cat.sh. If I run cat.sh by itself in a separate window/term, it runs as expected.

Does anyone have any ideas on how I can get these scripts to run in parallel?

Your current setup is running your commands sequentially. The first one runs, and completes, allowing the second one to start. The second one never ends, so the third one never gets a chance to start.

Try this:

foo.sh >foo.out 2>foo.err &

dog.sh >dog.out 2>dog.err &

cat.sh >cat.out 2>cat.err &

That will tell each command to run in the background, and send their output and error information to files. When a command is run in the background, it detaches from the current session, and allows the next command to run. It's important to put the ampersand (&) as the last character of the line. Otherwise, it might not get recognized correctly.

Alternately, have each of your shell scripts run the associated .php script you're apparently calling with the same semantics as above. In this scenario, your foo.sh might contain:

#!/bin/bash

php foo.php >foo.out 2>foo.err &

Finally, if you need true parallel operation, check out GNU Parallel.

Note: If you really want the output going to the screen/terminal, you can leave off the output redirection:

foo.sh &

Remember, though, that running multiple commands concurrently can cause confusing output (you may now know what output or errors came from which command), and if you logout, the output has nowhere to go.

This can be done with parallel command.

Beautiful examples: parallel

[me@neo]<bash><~> 19

05:51 Tue Mar 05 > man -f parallel

parallel (1) - run programs in parallel

--

parallel -j 3 -- "sh foo.sh" "sh dog.sh" "sh cat.sh"

This will start 3 scripts in parallel, at the same time.

parallel sh -c "echo hi; sleep 2; echo bye" -- 1 2 3

This will start by default as many process of the same command, depending of cpu count.

parallel -j 3 sh -c "echo hi; sleep 2; echo bye" -- 1 2 3

This will force to execute 3 processes at the same time, even if you have just 2 cpu's.

Example:

[me@neo]<bash><~> 31 

06:09 Tue Mar 05 > parallel -j 10 sh -c "echo Hello World; sleep 3; echo Good morning" -- $(seq 1 10)



Hello World

Hello World

Hello World

Hello World

Hello World

Hello World

Hello World

Hello World

Hello World

Hello World

Good morning

Good morning

Good morning

Good morning

Good morning

Good morning

Good morning

Good morning

Good morning

Good morning

Chris has the good general idea for most scripts: end each command with & and they'll go into the background. That said, here's my something I keep around for some higher-end tasks. Let me introduce you to my handy toolbox script, parallel-exec.pl

    #!/usr/bin/perl

    use threads;

    use Thread::Queue;



    my @workers;

    my $num_threads = shift;

    my $dbname = shift;

    my $queue = new Thread::Queue;



    for (0..$num_threads-1) {

            $workers[$_] = new threads(&worker);

                    print "TEST!n";

    }



    while ($_ = shift @ARGV) {

            $queue->enqueue($_);

    }



    sub worker() {

            while ($file = $queue->dequeue) {

                    system ('./parser.pl', $dbname, $file);

            }

    }



    for (0..$num_threads-1) { $queue->enqueue(undef); }

    for (0..$num_threads-1) { $workers[$_]->join; }

With some light tweaking, you can change the number of threads, provide an array of commands, etc. As this stands right now, it takes the number of threads specified on the command line, a database name, and a list of files to import, then spawns one instance for each file running the specified number at once.

or..you can do it like this:

threads=20

tempfifo=$PMS_HOME/$$.fifo



trap "exec 1000>&-;exec 1000<&-;exit 0" 2

mkfifo $tempfifo

exec 1000<>$tempfifo

rm -rf $tempfifo



for ((i=1; i<=$threads; i++))

do

    echo >&1000

done



for ((j=1; j<=1000; j++))

do

    read -u1000

    {

        echo $j

        echo >&1000

    } &

done



wait

echo "done!!!!!!!!!!"

every time, it runs 20 threads in parallel.

hope it help :)

Try this :
./Cat.sh &
./Dog.sh &
./foo.sh & they will run in background.

For more control on the processes embbed them on a python script using multiprocessing module

import multiprocessing as mp
import os
def Run(script_name):
os.system('./'+script _name+'.sh)
Pool=[]
For s in [your script list]:
p=mp.Process(target =Run ,args=(s,))
Pool.append(p)
For p in P:
p.start()
Fort p in P:
p.join()