Using pipes to eliminate temporary files

_README_replacing_temporary_files_with_pipes.md

Using pipes to eliminate temporary files

This illustrates using pipes to eliminate temporary files. We start with a bash script that takes some demographic data (see MOCK_DATA.csv) specified as a file name as a command line argument. The script then outputs a count of how many people come from different states. The output on the included data file is:

  49 MN
  21 IA
  20 WI
   7 ND
   3 SD

The first version generates a lot of temporary text files (one for each step); the second does the same thing but uses pipes (|) to turn the output of each command into the input of the next command. This avoids creating a ton of extra temporary files, each of which we have to name, and naming is hard. We should also delete each of the temporary files when we're done so we don't clutter up the world. Thus not having them is Very Nice.

Note that we need to escape the newline at the end of each line (including comments) by putting a backslash (\) at the end of the line. (The backslash needs to be the very last character.) This causes bash to "ignore" the newline and see this as one (very long) line. We could actually make it one lone long, but breaking it up like this is a lot more readable.

The mock data was generated using Mockaroo.

See this StackOverflow answer from Jonathan Leffler for more on the advantages of pipes vs temporary files.

count_states.sh

#!/usr/bin/env bash

# Takes some demographic data (see MOCK_DATA.csv below)
# specified as a file name as a command line argument.
# Outputs a count of how many people come from different
# states. The output on the data file below is:
#
#  49 MN
#  21 IA
#  20 WI
#   7 ND
#   3 SD 

data_file="$1"

# Get rid of the header line
tail -n +2 "$data_file" > no_header.txt
# Extract just the state column
# The `-F ','` tells `awk` to use `,` as the field separator.
# That's necessary here because the fields are separated by
# commas and not spaces, which is `awk`'s default field
# separator.
awk -F ',' '{ print $5 }' no_header.txt > just_states.txt
# Sort the states so I can use `uniq` to count
sort just_states.txt > sorted_states.txt
# Now count with uniq
uniq -c sorted_states.txt > state_counts.txt
# Sort by occurrances. `-n` tells `sort` to sort numerically
# (instead of alphabetically), and `-r` tells it to reverse
# the order so the biggest values end up at the top.
# This sends the output to standard output.
sort -nr state_counts.txt

count_states_with_pipes.sh

#!/usr/bin/env bash

# Takes some demographic data (see MOCK_DATA.csv below)
# specified as a file name as a command line argument.
# Outputs a count of how many people come from different
# states. The output on the data file below is:
#
#  49 MN
#  21 IA
#  20 WI
#   7 ND
#   3 SD 

# This is the same as before, but we use pipes (`|`) to turn the
# output of one command into the input of the next command. This
# avoids creating a ton of extra temporary files, each of which we
# have to name, and naming is hard. We should also delete each of the
# temporary files when we're done so we don't clutter up the
# world. Thus not having them is nice.

# We need to escape the newline at the end of each line (all the
# backslashes `\`) so the shell sees this as one (very long) line.
# We could actually make it one lone long, but this is a _lot_ more
# readable.

data_file="$1"

# Get rid of the header line
tail -n +2 "$data_file" | \
  # Extract just the state column \
  # The `-F ','` tells `awk` to use `,` as the field separator. \
  # That's necessary here because the fields are separated by \
  # commas and not spaces, which is `awk`'s default field \
  # separator. \
  awk -F ',' '{ print $5 }' | \
  # Sort the states so I can use `uniq` to count \
  sort | \
  # Now count with uniq \
  uniq -c | \
  # Sort by occurrances. `-n` tells `sort` to sort numerically \
  # (instead of alphabetically), and `-r` tells it to reverse \
  # the order so the biggest values end up at the top. \
  # This sends the output to standard output. \
  sort -nr 

mock_data.csv