pipeline_trick.md

A funky shell thingy that I've never seen before

So you're in posix sh and you want to do the equivalent of this in bash:

foo | tee >(bar) >(baz) >/dev/null

(Suppose that bar and baz don't produce output. Add redirections where needed if that's not the case.)

The "standard" way to do this would be to create the named pipes manually, and start bar and baz in background:

mkfifo fifo1 fifo2
bar < fifo1 &
baz < fifo2 &
foo | tee fifo1 fifo2 >/dev/null

This works, but it puts bar and baz in their own process groups. But that's terrible! It fucks up the signals and job control!

Signals are now only delivered to the foreground pipeline. Deadly signals are largely going to be handled correctly, as most well behaved programs terminate on EOF, but that's not guaranteed. And you can't suspend/unsuspend the whole thing anymore!

You can work around some of the issues with the signals, for instance by adding an exit trap and using it to forward any deadly signal to the bg processes:

trap 'kill "$pid1" "$pid2"' EXIT

mkfifo fifo1 fifo2
bar < fifo1 & pid1=$!
baz < fifo2 & pid2=$!
foo | tee fifo1 fifo2 >/dev/null

Similarly, you can just trap any signals and forward them manually.

That's cool, but now the signals come from the shell (you can see it in siginfo_t). You're also relying on the shell to stay alive long enough to deliver them, so if the signal was sigkill the shell couldn't forward it.

Instead, you can create the fifos manually, and then... use a pipeline!

mkfifo fifo1 fifo2
foo | tee fifo1 fifo2 >/dev/null |
bar < fifo1 |
baz < fifo2

Now all your processes are in the same process group, so any signals sent to the group get delivered to all the processes.

You can even handle cleaning up the fifos this way:

{ rm fifo1; bar; } < fifo1

This looks racy, because tee needs to open the fifos before rm removes them, but it's actually entirely safe because the shell is executing the redirection first, and opening the fifo in read mode blocks until there's at least one writer.

You can now safely ^C and forget.

---

Please let me know if this thing has a name. I don't know how to Google it, but surely someone came up with it like 40 years ago.

@oguz-ismail

(Suppose that bar and baz don't produce output. Add redirections where needed if that's not the case.)

You can just redirect if you need that output with the classic fd juggling trick:

{ bar < fifo1 >&4 | baz < fifo2; } 4>&1

@izabera In that case I'd do

{ { foo | tee /proc/self/fd/3 | bar >&4; } 3>&1 | baz; } 4>&1

but yeah, mkfifo is more portable than /proc/self/fd.