Monitoring pipes in Linux

Have you ever said

gzip file > /path/to/an/nfs/drive/file.gz

and wondered how far it’d gotten?  I did that just today, more or less.  I poked around for something that’d tell me how far gzip had gotten, and found pv (http://www.ivarch.com/programs/pv.shtml).

From the pv manpage:

pv allows a user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and ETA.

To use it, insert it in a pipeline between two processes, with the appropriate options.  Its standard input will be passed through to its standard output and progress will be shown on standard error.

pv will copy each supplied FILE in turn to standard output (- means standard input), or if no FILEs are specified just standard input is copied. This is the same behav‐ iour as cat(1).

A simple example to watch how quickly a file is transferred using nc(1):

pv file | nc -w 1 somewhere.com 3000

A similar example, transferring a file from another process and passing the expected size to pv:

cat file | pv -s 12345 | nc -w 1 somewhere.com 3000

A more complicated example using numeric output to feed into the dialog(1) program for a full-screen progress display:

(tar cf - . 
| pv -n -s $(du -sb . | awk '{print $1}')
| gzip -9 > out.tgz) 2>&1
| dialog --gauge 'Progress' 7 70

Frequent use of this third form is not recommended as it may cause the programmer to overheat.

I like that last line.  🙂

And so now I have

#pv a_large_file | gzip > /mnt/backup/a_large_file.gz
4.41GB 0:08:56 [12.2MB/s] [==> ] 31% ETA 0:19:09

Woohoo!