There's also `progress` which works for tools mainly operating on a single file, but unlike `pv`, you don't have to start the tool differently. It'd e.g. work nicely for the `gzip` example. Just call `progress` on a different terminal while `gzip` is running.
Pipe viewer? What's that? Let me check the post...oh, it's good old pv! Never noticed it had a full name, damn Unix utilities with their short names!
pv is great.
It has a limit parameter so you can limit the speed. Great if you don't want to saturate some link or have additional costs for uploading above a certain rate per hour/day.
Also useful for testing behaviour on slow filesystem / connections.
It can take a pid argument too, -d IIRC, which will get it to display progress info for all the open file descriptors of a running process.
Really useful as a quick way to check what a IO process is doing if appears to be stuck.
I like to use pv as a quick and dirty operations per second counter. Sometimes I will write a program or script that does a bunch of things in parallel (e.g. RPCs to a service I'm working on), and prints one line of output for every operation completed. Then I pipe that output to pv using the --lines option to count only lines. It shows how many lines are being printed per second, which roughly counts operations per second. (IIRC, also need to pipe to /dev/null to prevent pv's fancy output from clobbering the tool's output).
Fun stuff! Especially when combined with GNU parallel, in cases where the thing I'm measuring isn't already parallelized, and I want to be lazy.
I love pv but how much does adding the pipe affect overhead? I feel like most of my big jobs I want to measure are on things where you want the program to have direct access to the underlying file or storage. `pv somefile | dd` is going to be slower than `dd somefile`. At least I think so? I have no idea what modern Linux I/O can optimize.
Also does pv necessitate doing single threaded I/O?
> The obvious way to do it is:
> $ gzip -c access.log > access.log.gz
Is it?
Yes! My `,pv` is approximately: (probably a better way to make the k, but I stop once something is adequate; maybe I just need to make a `,,kof`)
tar cpS "$@" --sort=name | pv -bratpes $(du -cks "$@"|sed -n '/.total$/ s/.total$//p')k
,pv files/ | tar xp -C /destination
,pv files/ | ssh overthere tar xp -C /destinationA little more typing, but I find dd present on most systems already, so I tend to do this:
tar ... | dd status=progress | ...See also vipe(1) from the wonderful moreutils: https://manpages.debian.org/stretch/moreutils/vipe.1.en.html
We have that in powershell also show-progress
Got me wondering, how does it works?
Pipe viewer is excellent. I use it all the time.
As of version 1.8.10[1], which includes my merge request[2] to add an '--output' option, it has even completely replaced my use of 'dd' for writing disk images: 'sudo pv -Yo /dev/mmcblk0 whatever.img' is nicer, has much better progress indication, automatically selects a more sensible buffer size, and begets fewer groans from UNIX neckbeards, than the old 'sudo dd of=/dev/mmcblk0 if=whatever.img'. (The '-Y' causes pv to sync after each write, which greatly improves progress indication in Linux.)
Though it's useful for much more of course. I use it for progress when compressing files ('pv blah | gzip ...'), when uploading files to the web ('pv blah | curl --upload-file - ...' — curl doesn't show progress when uploading for whatever reason), or just when I wanna see that something is happening with an operation which would otherwise take a while (even things like a slow 'du -h /some/path | sort -h' benefits from a 'pv' squeezed in the middle just to indicate that something is happening).
[1] https://codeberg.org/a-j-wood/pv/releases/tag/v1.8.10
[2] https://codeberg.org/a-j-wood/pv/pulls/90