Skip to content

Instantly share code, notes, and snippets.

@balupton

balupton/000-readme.md

Last active February 11, 2025 22:45
Show Gist options
  • Save balupton/cd779f3a39507f75d5956a67e5543ab8 to your computer and use it in GitHub Desktop.
Save balupton/cd779f3a39507f75d5956a67e5543ab8 to your computer and use it in GitHub Desktop.
Download ZIP
Why does MacOS always append to a redirected file descriptor even when told to overwrite? Ubuntu only appends when strictly told to append
Raw
000-readme.md

Given the following code:

out="$(mktemp)"
rm -f "$out"
clear

printf '%s\n' 0 >"$out"
{
	printf '%s\n' '1' >/dev/stdout
	printf '%s\n' '2' >/dev/stdout
} >"$out"
cat -e -- "$out"
rm -f "$out"

On Ubuntu this outputs:

2$

On MacOS this outputs:

1$
2$

When explicitly appending, they behave consistently:

out="$(mktemp)"
rm -f "$out"
clear

printf '%s\n' 0 >"$out"
{
	printf '%s\n' '1' >/dev/stdout
	printf '%s\n' '2' >>/dev/stdout
} >"$out"
cat -e -- "$out"
rm -f "$out"

On MacOS and Ubuntu this outputs:

1$
2$

The most confusing example to me is this one:

out="$(mktemp)"
rm -f "$out"
clear

printf '%s\n' 0 >"$out"
exec 3>>"$out"
{
	printf '%s\n' '1' >/dev/stdout
	printf '%s\n' '2' >/dev/stdout
} >&3
{
	printf '%s\n' '3' >/dev/stdout
	printf '%s\n' '4' >/dev/stdout
} >&3
cat -e -- "$out"
rm -f "$out"
exec 3>&-

Which on MacOS outputs:

0$
1$
2$
3$
4$

Which on Ubuntu outputs:

4$

I was expecting this on Ubuntu:

0$
2$
4$

I am thoroughly confused why this behaviour occurs, in this example, and all the other examples I've devised to illustrate this discrepancy.

My questions:

  • What is this descrepancy? What is happening? Is this descrepancy intentional?
  • Where else does this descrepancy apply? What is its origins?
  • If this descrepancy is intentional, why was it justified? Which one should be the correct behaviour?
  • What can be done to mitigate these differences when writing cross-OS scripts?
  • Is shopt -o noclobber the appropriate response? Is this the true necessity of noclobber?

Cross-posted:

Raw
03-the-core-issue
#!/usr/bin/env bash
in='first non-empty line
second non-empty line
third non-empty line after blank line'
out="$(mktemp)"
rm -f "$out"
clear
echo "=== $LINENO === only second line ==="
function reader {
while read -r line; do
if [[ -z $line ]]; then
break
fi
printf '[%s]\n' "$line" >"$out"
done
}
reader <<<"$in"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === only second line ==="
function reader {
while read -r line; do
if [[ -z $line ]]; then
break
fi
printf '[%s]\n' "$line" >/dev/stdout
done
}
reader <<<"$in" >>"$out"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === first and second lines ==="
function reader {
local lines=()
while read -r line; do
if [[ -z $line ]]; then
break
fi
lines+=("$line")
done
printf '[%s]\n' "${lines[@]}" >/dev/stdout
}
reader <<<"$in" >"$out"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === 1 and 2 ==="
{
printf '%s\n' '1'
printf '%s\n' '2'
} >"$out"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === only 2 ==="
{
printf '%s\n' '1' >/dev/stdout
printf '%s\n' '2' >/dev/stdout
} >"$out"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === 1 and 2 ==="
printf '%s\n' 0 >"$out"
exec 3>"$out"
{
printf '%s\n' '1' >&3
printf '%s\n' '2' >&3
}
cat -e -- "$out"
rm -f "$out"
exec 3>&-
echo "=== $LINENO === 0 and 1 and 2 ==="
printf '%s\n' 0 >"$out"
exec 3>>"$out"
{
printf '%s\n' '1' >&3
printf '%s\n' '2' >&3
}
cat -e -- "$out"
rm -f "$out"
exec 3>&-
echo "=== $LINENO === 1, 2, 3, 4 ==="
printf '%s\n' 0 >"$out"
exec 3>"$out"
{
printf '%s\n' '1' >&3
printf '%s\n' '2' >&3
}
{
printf '%s\n' '3' >&3
printf '%s\n' '4' >&3
}
cat -e -- "$out"
rm -f "$out"
exec 3>&-
echo "=== $LINENO === 1, 2, 3, 4 ==="
printf '%s\n' 0 >"$out"
exec 3>"$out"
{
printf '%s\n' '1'
printf '%s\n' '2'
} >&3
{
printf '%s\n' '3'
printf '%s\n' '4'
} >&3
cat -e -- "$out"
rm -f "$out"
exec 3>&-
echo "=== $LINENO === only 4 ==="
printf '%s\n' 0 >"$out"
exec 3>"$out"
{
printf '%s\n' '1' >/dev/stdout
printf '%s\n' '2' >/dev/stdout
} >&3
{
printf '%s\n' '3' >/dev/stdout
printf '%s\n' '4' >/dev/stdout
} >&3
cat -e -- "$out"
rm -f "$out"
exec 3>&-
# `} >>&3` and `>> >&3` are invalid syntax
echo "=== $LINENO === only 4 === why is this only 4!!!! it should be 0, 2, 4... right.... right....?!"
printf '%s\n' 0 >"$out"
exec 3>>"$out"
{
printf '%s\n' '1' >/dev/stdout
printf '%s\n' '2' >/dev/stdout
} >&3
{
printf '%s\n' '3' >/dev/stdout
printf '%s\n' '4' >/dev/stdout
} >&3
cat -e -- "$out"
rm -f "$out"
exec 3>&-
echo "=== $LINENO === 1 and 2 ==="
printf '%s\n' 0 >"$out"
exec 3>"$out"
exec 4<"$out"
{
printf '%s\n' '1' >&3
printf '%s\n' '2' >&3
}
cat -e <&4
rm -f "$out"
exec 3>&- 4>&-
echo "=== $LINENO === 1 and 2 ==="
printf '%s\n' 0 >"$out"
{
printf '%s\n' '1' >/dev/stdout
printf '%s\n' '2' >>/dev/stdout
} >"$out"
cat -e -- "$out"
rm -f "$out"
# on macos >/dev/stdout is equivalent to >&1, which then itself is redirected to >"$out"
# on linux >/dev/stdout is equivalent to whatever stdout was redirected to >"$out"
# this means on macos: { echo 1 >/dev/stdout; echo 2 >/dev/stdout; } >"$out"
# is the same as: { { echo 1 >&1; echo 2 >&1; } >&1; } >"$out"
# whereas on linux it becomes: { echo 1 >"$out"; echo 2 >"$out"; }
echo "=== $LINENO === 1 and 2 ==="
printf '%s\n' 0 >"$out"
{
{
printf '%s\n' '1'
printf '%s\n' '2'
} >/dev/stdout
} >"$out"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === only 2 ==="
printf '%s\n' 0 >"$out"
{
{
printf '%s\n' '1' >/dev/stdout
printf '%s\n' '2' >/dev/stdout
} >/dev/stdout
} >"$out"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === only 2 ==="
printf '%s\n' 0 >"$out"
{
{
printf '%s\n' '1' >/dev/stdout
printf '%s\n' '2' >/dev/stdout
} >&1
} >"$out"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === 1 and 2 ==="
printf '%s\n' 0 >"$out"
{
printf '%s\n' '1' >&1
printf '%s\n' '2' >&1
} >"$out"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === 1 and 2 ==="
printf '%s\n' 0 >"$out"
{
{
printf '%s\n' '1' >&1
printf '%s\n' '2' >&1
} >/dev/stdout
} >"$out"
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === only 2 ==="
printf '%s\n' 0 >"$out"
# order of >&1 doesn't matter
{
printf '%s\n' '1' >/dev/stdout
printf '%s\n' '2' >/dev/stdout
} >"$out" >&1
cat -e -- "$out"
rm -f "$out"
echo "=== $LINENO === 1 and 2 ==="
printf '%s\n' 0 >"$out"
{
printf '%s\n' '1' '2' >/dev/stdout
} >"$out"
cat -e -- "$out"
rm -f "$out"
# bash 4.1 and above
echo "=== $LINENO === nothing === CUSTOM FD BIDIRECTIONAL ==="
exec {CUSTOM_FD}<>"$out"
printf '[%s][%s]\n' ${CUSTOM_FD} ${CUSTOM_FD} >/dev/tty
{
printf '%s\n' '1' >&${CUSTOM_FD}
printf '%s\n' '2' >&${CUSTOM_FD}
}
cat -b <&${CUSTOM_FD}
rm -f "$out"
exec {CUSTOM_FD}>&-
# bash 4.1 and above
echo "=== $LINENO === 1 and 2 === CUSTOM FD READ AND WRITE ==="
exec {CUSTOM_WRITE}>"$out"
exec {CUSTOM_READ}<"$out"
printf '[%s][%s]\n' ${CUSTOM_WRITE} ${CUSTOM_WRITE} >/dev/tty
{
printf '%s\n' '1' >&${CUSTOM_WRITE}
printf '%s\n' '2' >&${CUSTOM_WRITE}
}
cat -b <&${CUSTOM_READ}
rm -f "$out"
exec {CUSTOM_WRITE}>&- {CUSTOM_READ}>&-
# bash 4.1 and above
echo "=== $LINENO === 1 2 3 4 === CUSTOM FD READ AND WRITE - APPENDING ==="
exec {CUSTOM_WRITE}>>"$out" # appending
exec {CUSTOM_READ}<"$out"
printf '[%s][%s]\n' ${CUSTOM_WRITE} ${CUSTOM_WRITE} >/dev/tty
{
printf '%s\n' '1' >&${CUSTOM_WRITE}
printf '%s\n' '2' >&${CUSTOM_WRITE}
}
{
printf '%s\n' '3' >&${CUSTOM_WRITE}
printf '%s\n' '4' >&${CUSTOM_WRITE}
}
cat -b <&${CUSTOM_READ}
rm -f "$out"
exec {CUSTOM_WRITE}>&- {CUSTOM_READ}>&-
Raw
03-the-core-issue--macos-result.txt
=== 12 === only second line ===
[second non-empty line]$
=== 26 === only second line ===
[first non-empty line]$
[second non-empty line]$
=== 40 === first and second lines ===
[first non-empty line]$
[second non-empty line]$
=== 56 === 1 and 2 ===
1$
2$
=== 64 === only 2 ===
1$
2$
=== 73 === 1 and 2 ===
1$
2$
=== 85 === 0 and 1 and 2 ===
0$
1$
2$
=== 97 === 1, 2, 3, 4 ===
1$
2$
3$
4$
=== 113 === 1, 2, 3, 4 ===
1$
2$
3$
4$
=== 129 === only 4 ===
1$
2$
3$
4$
=== 146 === only 4 === why is this only 4!!!! it should be 0, 2, 4... right.... right....?!
0$
1$
2$
3$
4$
=== 162 === 1 and 2 ===
1$
2$
=== 175 === 1 and 2 ===
1$
2$
=== 189 === 1 and 2 ===
1$
2$
=== 201 === only 2 ===
1$
2$
=== 212 === only 2 ===
1$
2$
=== 223 === 1 and 2 ===
1$
2$
=== 232 === 1 and 2 ===
1$
2$
=== 243 === only 2 ===
1$
2$
=== 253 === 1 and 2 ===
1$
2$
Raw
03-the-core-issue--ubuntu-result.txt
=== 12 === only second line ===
[second non-empty line]$
=== 26 === only second line ===
[second non-empty line]$
=== 40 === first and second lines ===
[first non-empty line]$
[second non-empty line]$
=== 56 === 1 and 2 ===
1$
2$
=== 64 === only 2 ===
2$
=== 73 === 1 and 2 ===
1$
2$
=== 85 === 0 and 1 and 2 ===
0$
1$
2$
=== 97 === 1, 2, 3, 4 ===
1$
2$
3$
4$
=== 113 === 1, 2, 3, 4 ===
1$
2$
3$
4$
=== 129 === only 4 ===
4$
=== 146 === only 4 === why is this only 4!!!! it should be 0, 2, 4... right.... right....?!
4$
=== 162 === 1 and 2 ===
1$
2$
=== 175 === 1 and 2 ===
1$
2$
=== 189 === 1 and 2 ===
1$
2$
=== 201 === only 2 ===
2$
=== 212 === only 2 ===
2$
=== 223 === 1 and 2 ===
1$
2$
=== 232 === 1 and 2 ===
1$
2$
=== 243 === only 2 ===
2$
=== 253 === 1 and 2 ===
1$
2$
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment