Tuesday, April 3, 2012

Make parallel script in bash

I have a script what does thing as processing a file line-by-line. I modified this script to do parallel processing like follows:

#!/bin/bash

MAXJOBS=3

while read line; do
( echo "do thing with $line" ) &

while [ "`jobs -p | wc -l`" -ge "$MAXJOBS" ]; do
sleep 1;
done

done < input.txt

wait

It starts the sub processes and if the count is as great as MAXJOBS then waits. At the end of the script the wait waits all the remaining processes.

UPDATE: If you need som for example grepping before on the input you should use:


...
done < <(cat input.txt|grep '#')

wait

instead of

cat input.txt|grep '#'| while read line; do

2 comments:

.pQd said...

or - a bit simpler - you can use xargs with -P -n parameters.

samvittighed said...

Why not just use GNU Parallel?

cat input.txt|grep '#'| parallel do stuff

https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1