Reference: Bash Redirection

Bash Redirection

Before a command is executed, its input and output may be redirected using a special notation interpreted by the shell. Redirection allows commands’ file handles to be duplicated, opened, closed, made to refer to different files, and can change the files the command reads from and writes to. Redirection may also be used to modify file handles in the current shell execution environment. The following redirection operators may precede or appear anywhere within a simple command or may follow a command. Redirections are processed in the order they appear, from left to right.

Each redirection that may be preceded by a file descriptor number may instead be preceded by a word of the form {varname}. In this case, for each redirection operator except >&- and <&-, the shell will allocate a file descriptor greater than 10 and assign it to {varname}. If >&- or <&- is preceded by {varname}, the value of varname defines the file descriptor to close. If {varname} is supplied, the redirection persists beyond the scope of the command, allowing the shell programmer to manage the file descriptor’s lifetime manually.

In the following descriptions:

If the file descriptor number is omitted, and the first character of the redirection operator is <, the redirection refers to the standard input (file descriptor 0).
If the first character of the redirection operator is >, the redirection refers to the standard output (file descriptor 1).

The word following the redirection operator in the following descriptions, unless otherwise noted, is subjected to brace expansion, tilde expansion, parameter expansion, command substitution, arithmetic expansion, quote removal, filename expansion, and word splitting. If it expands to more than one word, Bash reports an error.

Bash Redirection: I/O Order

Bash redirections are:

Processed from left to right,
Interpreted iteratively;

Note that the order of redirections is significant. For example, the command:

ls > dirlist 2>&1
# directs both standard output (file descriptor 1) and standard error (file descriptor 2) to the file dirlist

..while the command:

ls 2>&1 > dirlist
# directs only the standard output to file dirlist, because the standard error was made a copy of the standard 
# output before the standard output was redirected to dirlist.

An earlier redirection can affect a later one:

If an earlier redirection has redirected a given stream (identified by a file descriptor number, such as 1 for stdout (the default), and 2 for stderr), later redirections targeting that stream refer to the already-redirected version.

..but not vice versa.

A later redirection has no retroactive effect on the the target of an earlier redirection:
- If you specify file descriptor 1 as the target in an earlier redirection, what 1 means at that time is locked in, even if 1 is redirected later.
- Note: Output isn't actually sent until all redirections are in place, and that any redirection-target output files are created or truncated before command execution begins (this is the reason why you can't read from and redirect output to the same file with a single command).

Bash handles several filenames specially when they are used in redirections, as described in the following table. If the operating system on which Bash is running provides these special files, bash will use them; otherwise it will emulate them internally with the behavior described below.

/dev/fd/fd
# If fd is a valid integer, file descriptor fd is duplicated.

/dev/stdin
# File descriptor 0 is duplicated.

/dev/stdout
# File descriptor 1 is duplicated.

/dev/stderr
# File descriptor 2 is duplicated.

/dev/tcp/host/port
# If host is a valid hostname or Internet address, and port is an integer port number or service name, 
# Bash attempts to open the corresponding TCP socket.

/dev/udp/host/port
# If host is a valid hostname or Internet address, and port is an integer port number or service name, 
# Bash attempts to open the corresponding UDP socket.

A failure to open or create a file causes the redirection to fail.

Redirections using file descriptors greater than 9 should be used with care, as they may conflict with file descriptors the shell uses internally.

Bash Redirection: Questions & Answers

Q: By default, a Unix program has 3 inputs / outputs. What are they?

stdin  # stdin: for input, like from a pipe
stdout # stdout: for regular output
stderr # stderr: an output for debugging information, errors, and any informational messages meant for a human to read

Q: When you run a program from a terminal, where does `stdout` output to?

the terminal!
stdin, stderr, and stdout all use the terminal by default

Q: How do you redirect a program's output to a file?

$ some_cmd > file.txt

Q: How do you redirect a program's error output to a file?

$ some_cmd 2> file.txt

Q: How do you set a program's input (`stdin`) to be a file instead of the terminal?

$ some_cmd < file.txt

Q: What's a way to ignore a program's output?

# /dev/null is a special file. The operating system completely ignores all writes to /dev/null.
$ some_cmd > /dev/null

Q: Why is the number `2` in `$ ls 2> file.txt` `2` and not `4` or `7` or `13`?

It's the file descriptor number.
Every Unix program has a number called a "file descriptor".
The first 3 numbers are generally reserved for stdin/stdout/stderr:

0 # for stdin
1 # for stdout
2 # for stderr

Q: How do you redirect `stderr` output to `stdout`?

$ some_cmd 2>&1

# example, to grep error output:
$ strace ls 2>&1 | grep whatever

Q: How do you redirect `stdout` and `stderr` to a file?

# This one always looks weird to me, but it's the only one that works.
$ some_cmd > file 2>&1

Q: Will `sudo echo x > /file` allow you to redirect to `/file` if it's owned by `root`?

# Nope. It doesn't work because bash opens /file to manage the redirect before echo starts, 
# and bash is running as you. This works, though:

echo x | sudo tee /file
# tee copies everything from its standard input to stdout and a file. 
# Because tee is responsible for opening the file (and not bash), sudo tee will be able open /file as root

Bash Redirection: Other Examples

Don't truncate when redirecting

A common problem in shell languages is truncating a file that you’re trying to transform by redirecting into it.

Let’s say I have a file full of a‘s

$ cat test.txt
aaaaaa

And I want to switch them all to b‘s

$ cat test.txt | tr 'a' 'b'
bbbbbb

But when I try to redirect into the same file I truncate file:

$ cat test.txt | tr 'a' 'b' > test.txt
$ cat test.txt
# nothing

This is because the shell truncates “test.txt” for redirection before cat reads the file.

An alternate approach is to use tee.

$ cat test.txt | tr 'a' 'b' | tee test.txt
# bbbbbb
# See: https://explainshell.com/explain?cmd=cat+test.txt+%7C+tr+%27a%27+%27b%27+%7C+tee+test.txt

$ cat test.txt
# bbbbbb

tee will write the output to both the file and to standard out, but will do so at the end of the pipe, after the cat command reads from the file.

Bash Redirection: Cheat Sheet

$ python hello.py > output.txt   # stdout to (file)
$ python hello.py >> output.txt  # stdout to (file), append
$ python hello.py 2> error.log   # stderr to (file)
$ python hello.py 2>&1           # stderr to stdout
$ python hello.py 2>/dev/null    # stderr to (null)
$ python hello.py &>/dev/null    # stdout and stderr to (null)
$ python hello.py < foo.txt      # feed foo.txt to stdin for python
$ diff <(ls -r) <(ls)            # Compare two stdout without file

JoeyBurzynski/Reference: Bash Redirection - How to Use Bash Redirection.md