Parallel UNIX utilities

This post, to which I will add to over time, covers some usefull Unix shell utilities to run parallel processes on the command line.

Last modified: Jan 16, 2024

GNU Parallel

Fantastic tool to run shell command lines in parallel.

Here is a quick example taken form this list of examples. The following command will compress all HTML files in the current folder, 4 at a time. This means each process will run on a single core but 4 gzip processes will be started.

parallel -j4 gzip --best ::: *.html

On MacOS, you can install it with Homebrew: brew install parallel

I previously talked about parallel in my Efficient log processing blog post.

Parallel gzip: pigz

This utility is a drop-in replacement for gzip that compresses in parallel.

Contrastring with the previous command, this utility compresses one file at a time but using many cores:

pigz -p=4 --best index.html

Note that it also supports the -d flag to decompress.

On MacOS, you can install it with Homebrew: brew install pigz

Parallel sort: parsort

Drop-in replacement for gsort with the addition of the --parallel=n flag.

Source: https://superuser.com/a/1616124