Introduction
A modern server is typically multi-core, perhaps even multi-CPU. That is plenty of computing power to unleash on a given job. However, unless you run a job in parallel, you are not maximizing the use of all that power.
Below are some typical everyday operations we can speed up using parallel computing:
To execute a job in parallel, you can use any of the following commands:
Installation of GNU Parallel
To install
My objective is to run a shell command in parallel, but on the same multi-core machine. The command can take multiple options, but only 1 is variable. Specifically, you run concurrent instances of the command by providing a different value for that one variable option. The different values are fed, one per line, to
The rest of this post shows how
Each input line is a directory location, e.g.,
Below are some typical everyday operations we can speed up using parallel computing:
- Backup files from multiple source directories to a removable disk.
- Resize image files in a directory.
- Compress files in a directory.
ppsspexecGNU parallel
GNU parallel command.Installation of GNU Parallel
GNU parallel on a Debian/Ubuntu system, run the following command:$ sudo apt-get install parallel
General Usage
TheGNU parallel program provides many options which you can specify to customize its behavior. Interested readers can read its man page to learn more about their usage. In this post, I will narrow the execution of GNU parallel to the following scenario.My objective is to run a shell command in parallel, but on the same multi-core machine. The command can take multiple options, but only 1 is variable. Specifically, you run concurrent instances of the command by providing a different value for that one variable option. The different values are fed, one per line, to
GNU parallel via the standard input.The rest of this post shows how
GNU parallel can backup multiple source directories by running rsync in parallel.Parallel backup
The following command backs up 2 directories in parallel:/home/peter and /data.$ echo -e '/home/peter\n/data' | parallel -j-2 -k --eta rsync -R -av {} /media/myBKUP
Standard input
Theecho command assembles the 2 source directory locations, separated by a newline character (\n), and pipes it to GNU parallel.How many jobs?
By default,GNU parallel deploys 1 job per core. You can override the default usint the -joption.-j specifies the maximum number of parallel jobs that GNU parallel can deploy. The maximum number can be specified in 1 of several ways:-jfollowed by a number-j2means that up to 2 jobs can run in parallel.-j+followed by a number-j+2means that the maximum number of jobs is the number of cores plus 2.-j-followed by a number-j-2means that the maximum number of jobs is the number of cores minus 2.
$ parallel --number-of-cores
8Keeping output order
Each job may output lines to the standard output. When multiple jobs are run in parallel, the default behavior is that a job's output is displayed as soon as the job finishes. You may find this confusing because the output order may be different from the input order. The-k option keeps the output sequence the same as the input sequence.Showing progress
The--eta option reports progress while GNU parallel executes, including the estimated remaining time (in seconds).Input place-holder
GNU parallel substitutes the {} parameter with the next line in the standard input.Each input line is a directory location, e.g.,
/home/peter. Instead of the full location, you can specify other parameters in order to extract a portion thereof - e.g., the directory name(/home) and the basename (peter). Please refer to the man page for details.Summary
GNU parallel is a tool that Linux administrators should add to their repertoire. Running a job in parallel can only improve one's efficiency. If you are already familiar with xargs, you will find the syntax familiar. Even if you are new to the command, there is a wealth of on-line help on the GNU parallel website.
