Make parallel script in bash
I have a script what does thing as processing a file line-by-line. I modified this script to do parallel processing like follows:
#!/bin/bash
MAXJOBS=3
while read line; do
( echo "do thing with $line" ) &
while [ "`jobs -p | wc -l`" -ge "$MAXJOBS" ]; do
sleep 1;
done
done < input.txt
wait
It starts the sub processes and if the count is as great as MAXJOBS then waits. At the end of the script the wait waits all the remaining processes.
UPDATE: If you need som for example grepping before on the input you should use:
...
done < <(cat input.txt|grep '#')
wait
instead of
cat input.txt|grep '#'| while read line; do
2 comments:
or - a bit simpler - you can use xargs with -P -n parameters.
Why not just use GNU Parallel?
cat input.txt|grep '#'| parallel do stuff
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Post a Comment